DSCI 100 Week 2 : Reading in data locally and from the web

Sky (Kehan) Sheng

Slides for Section 008

Uniform Resource Locator (URL): www.skysheng.io/dsci_100_sky_slides/

🌐 File Locations: Remote (on the Internet)

A remote file lives on the internet, and is accessed via a URL that starts with http:// or https://. For example:

http://traffic.libsyn.com/mbmbam/MyBrotherMyBrotherandMe367.mp3

📋 Housekeeping: Late Registrants

• Welcome! 🎉
• Any assignment from a class that happened before you registered is due 1 week after you registered
• Any assignment from a class that happened after you registered is due at the regular time

📋 Housekeeping: Communications

• Questions about course material, worksheets, or clarifications → Piazza or Office Hours
• Questions about academic concessions, due dates, late registration, admin stuff → Julia courses@stat.ubc.ca
• Private matters, accommodations, mentorship → email me at skysheng@cs.ubc.ca or come by Office Hours
• Office hours (see Canvas for schedule): feel free to drop by any office hours, regardless of the section you’re in!

📋 Housekeeping: A Few Reminders

• ❓ Questions are always welcome! Asking a question is a sign that you’re thinking. If you have a question, chances are others are wondering the same thing too
• 📖 Please read the book! the answers to your assignment questions are often already there
• 🌙 TAs & skys don’t necessarily work late nights or weekends. Start your assignments early: if you get stuck, it’s much quicker to hear back from the teaching team during normal working hours on weekdays

📋 Housekeeping: Worksheets/Tutorials

• No need to submit anything on Canvas — the server automatically submits your work for you at the deadline!

• Remember: when in doubt, Kernel → Restart Kernel and Run All (or the ▶▶ button)

• When working on assignments, you’ll see cells that say:

  DOUBLE CLICK TO EDIT THIS CELL AND REPLACE THIS TEXT WITH YOUR ANSWER

  ⚠️ Make sure you double click and edit that cell directly — your answer won’t be graded otherwise!

• 🔌 Shut down the server when you’re done working

Week 1 Recap

Different file formats

Hand-written notes

Excel sheet

• Hand-written notes: painful to read, slow to process, impossible to analyze at scale
• Excel sheets: an upgrade: colors, fonts, formatting, multiple sheets, but still manually edited, not reproducible, hard to analyze at scale, and capped at about 1 million rows
• Reminder: each row = one observation, each column = one variable (a characteristic of that observation)

More file formats

animal,legs,sound otter,4,squeak owl,2,hoot duck,2,quack Comma-Separated Values (CSV)	animal legs sound otter 4 squeak owl 2 hoot duck 2 quack Tab-Separated Values (TSV)
animal;legs;sound otter;4;squeak owl;2;hoot duck;2;quack Semicolon-Separated Values (SSV)	animal|legs|sound otter|4|squeak owl|2|hoot duck|2|quack …or anything you want, just be consistent!

• CSV: a plain text file — small, easy to read/process/share across computers, and readable by R and other languages for analysis at scale

Different way to “bag” and “read” data

• File formats are just different “bags” for storing and carrying around the same data

What is R package? What is a function? What is an argument?

Ask students to guess the answer based on the analogy image before revealing the next slide.

R package ≈ A box of tools; Function ≈ A tool to use; Argument ≈ The input to the tool

• Gift box = R package — a collection of tools (functions) someone else already built for you
• Coffee machine = function — a tool that does one job, e.g. read_csv() reads in a CSV file for you
• You don’t need to understand how it works internally to use it (just like using a laptop without knowing how to build one, that’s abstraction)
• Buttons (milk / water / coffee) = arguments — the inputs you choose to tell the function exactly what to do, e.g. telling read_csv() which file to read or what delimiter to use
• Cups coming out = output — what you get back after the function runs, depending on which argument(s) you picked

Writing Good Variable Names

• Can include letters (A–Z, a–z), numbers, underscores (_), and dots (.)
• Cannot start with a number (e.g., 2cats ❌)
• Cannot contain spaces — use my_cat, not my cat ❌
• Case-sensitive: Result and result are two different variables
• Use lowercase letters: my_age ✅
• Use underscores to separate words: favorite_color ✅
• Choose meaningful, descriptive names: student_count ✅ (not a or b ❌)

• The first four bullets are hard rules — break them and R will throw an error before your code even runs
• The last three are style conventions, not rules: a <- 42 runs perfectly fine, but good luck remembering what a means a week from now, or explaining it to a teammate

🆘 Frequently Asked Questions (FAQ)

• Error: object 'my_variable' not found → Happens when you run cells out of order in Jupyter. Run the code cell that creates the variable first, then check your spelling.

• Unmatched parentheses or quotes → Every ( needs a ), and every opening " needs a closing ".

• What does # do? → # is a symbol to tell R that you are writing a comment for humans to read. R ignores everything after it on that line. Comments are not instructions for R.

• Does empty space and indentation matter? → No,empty space and indentation are ignored by R.

  # This line is a comment -- R will not run it
  my_variable <- 10  # you can also add a comment after code

• Important: always click “Restart Kernel and Run All” before submitting, to make sure your notebook runs top-to-bottom without errors
• Comments leave a trail of breadcrumbs for yourself (and others) explaining why the code does what it does

🆘 FAQ: Should I use <- or = ?

• Use <- to assign a value to a variable (standard convention)

• Reserve = for setting arguments inside a function call.

  variable_name <- 2
  my_data <- read_csv("data/my_observation.csv", skip = 3)

• Why does <- vs = matter? → To avoid ambiguity, reserve <- for assignment and = for function arguments.

  mean(x = c(1, 2, 3))   # calculates the mean of 1, 2, 3; x is just an argument name
  mean(x <- c(1, 2, 3))  # creates a variable x storing c(1, 2, 3)

• variable_name = 2 technically works the same as variable_name <- 2 and successfully creates the variable, but there are cases where the two are not interchangeable, which is exactly what the next slide shows
• In the example: the first line calculates the average of 1, 2, 3 and doesn’t create anything new. The second line creates a variable x storing (1, 2, 3), and also returns their mean. Same-looking code, two very different side effects

🆘 FAQ: When should I use quotation marks?

• Use quotes ("...") around character strings (text).

• Variable and object names are never quoted.

• Quotation marks "" is letting R know that you want to print exactly that sentence or words into the output or plot.

  name <- "Sky"   # "Sky" is a string -> needs quotes
  print(name)     # name is a variable -> no quotes

• Quotation marks mark a beginning and an end: everything between the opening and closing " is treated as literal text, not as code R should evaluate

Functions You’ll Be Using from Tidyverse

• filter() — keep rows that match a condition
• select() — keep specific columns
• arrange() — sort rows by a column’s values
• slice() — keep rows by their position
• mutate() — create or modify a column
• ggplot() — build a plot, layer by layer

iClicker: Which time slot works best for you for Office Hours?

• (A) Tuesday 4–5pm
• (B) Thursday 4–5pm
• (C) Friday 11am–12pm
• (D) Friday 1–2pm

Week 2: Reading in data into R

🤔 Why Use R?

… and why can’t I just use Excel?

vs. 

There are many advantages to using a programming language like R (or Python, Julia, etc.) over spreadsheet software:

• statistical analysis functions that go beyond Excel
• free and open-source
• transparent & reproducible code
• can handle large amounts of data and complex analyses

• Excel: flexible, but limited to about a million rows and edited manually
• If your data has issues (negative values, missing values, etc.) and you fix them by hand, it’s easy to lose track of what you changed, how, and why

🍪 Why Use R? Think of It Like Baking

Using a programming language is like baking with a recipe:

Ingredients = data

Recipe = code

• Someone else can use your recipe (code) to bake the same cake (produce the same data analysis): that’s reproducibility!
• Spreadsheets in Excel make it very difficult to understand where results came from, since clicking and dragging around leaves no trace of what you did

🔄 Data Science Workflow

📥 Loading/Importing Data

4 most common ways to do this in data science:

1. read in a text file with data in a spreadsheet format
2. read from a database (e.g., SQLite, PostgreSQL)
3. scrape data from the web (optional bonus material)
4. use a web API (Application Programming Interface) to read data from a website (optional bonus material)

💻 File Locations: Local (on Your Computer)

• An absolute path locates a file with respect to the “root” folder on a computer
  • starts with /
  • e.g. /home/sky/documents/timesheet.xlsx
• A relative path locates a file relative to your working directory
  • doesn’t start with /
  • e.g. documents/timesheet.xlsx (where the working directory is /home/sky/)

📂 Same Folder Structure, Two Views

File Explorer (OS GUI)

.../
├── ... # other folders and files
└── story_of_otter/
    ├── clown_dungeon/
    │   ├── creature_david.txt
    │   ├── monster_bob.txt
    │   └── ollie.txt
    ├── home_of_ollie/
    └── ocean/

Directory Tree

Both panels show the exact same folder structure. The file explorer (left) is what you normally click through in Finder/File Explorer. The directory tree (right) shows the same nesting relationships in a compact, text-based way, often easier to read at a glance, and exactly how we’ll draw folder structures for the rest of this course.

🦦 Absolute vs. Relative Paths: story_of_otter Example

home/
├── isaac/
└── sky/                    
    ├── images/
    └── story_of_otter/ ⬅️ cwd
        ├── clown_dungeon/
        │   ├── creature_david.txt
        │   ├── monster_bob.txt
        │   └── ollie.txt
        ├── home_of_ollie/
        └── ocean/

absolute path	relative path
/home/sky/story_of_otter/clown_dungeon	clown_dungeon
/home/sky/story_of_otter/clown_dungeon/ollie.txt	clown_dungeon/ollie.txt
/home/sky/images	../images
/home/sky/story_of_otter	.
/home/sky	~
/home/sky/story_of_otter/ocean	~/story_of_otter/ocean

cwd (current working directory) = /home/sky/story_of_otter.

Row 1: a folder directly inside cwd — the relative path is just its name.

Row 2: a file nested one level deeper — the relative path chains the folder and file name together.

Row 3: ../ moves up one level from cwd before going into images, which lives next to story_of_otter, not inside it.

Row 4: . always refers to the current working directory itself.

Row 5: ~ is shorthand for the home directory, no matter what the cwd currently is.

Row 6: ~ can be combined with a longer relative path to point anywhere under the home directory, regardless of cwd.

📂 Absolute vs. Relative Paths

• Generally, to ensure your code can run on a different computer, you should use relative paths

• Example:

  • Alice is working inside the folder /home/alice/project/
  • To load data in her Jupyter notebook, she uses the absolute path /home/alice/project/data/happiness_report.csv
  • 🤔 What issue will arise when she shares the notebook with her collaborator Keeran, who tries to read in the data on their own computer?

• Keeran likely doesn’t have a folder called alice on their computer, so the absolute path /home/alice/... won’t exist — the code will fail
• Time permitting: demo the getwd() function

🔬 Demo: Loading Data Locally

Workflow:

1. Navigate to the file and see what it looks like
2. Prepare to load the data into R — might need to load a library, download a file, or connect to a database
3. Load the data into R — using read_csv(), read_delim(), etc.
4. Inspect the result to make sure it worked!

Let’s load the Old Faithful geyser dataset from Larry Wasserman’s book All of Statistics 🌋

🔬 Demo: Loading Data Locally

First, let’s navigate to the file and see what it looks like

• View the original file: https://www.stat.cmu.edu/~larry/all-of-statistics/=data/faithful.dat
• What’s wrong with this data? Long text header at the top, first column has no name, no clear separator between values
• Other ways data can be dirty: spelling mistakes, inconsistent number of columns
• Data cleaning is 80–90% of a data scientist’s work — always inspect your data before writing code (domain experts help too!)
• Note: JupyterHub is remote, so it can’t access files on your local computer directly, that’s why we download the file first
• Tip: right-click the file in the file browser to open it in a plain text editor and see what it looks like

🔬 Demo: Loading Data Locally

We’ll build up the read_delim() call step by step, fixing one problem at a time.

Step 1 — start simple, using a single whitespace ' ' as the delimiter:

read_delim('data/faithful.dat', delim = ' ')

Step 2 — skip the header. The first 25 lines are comments, so skip them:

read_delim('data/faithful.dat', delim = ' ', skip = 25)

Step 3 — fix the columns. The number of columns isn’t identified correctly, so supply column names explicitly (splitting the call across multiple lines for readability):

read_delim(
    'data/faithful.dat',
    delim = ' ',
    skip = 25,
    col_names = c('index', 'eruption_time', 'waiting_time')
)

Step 4 — skip one more line. The old header row is now being read as data, so skip one more line:

read_delim(
    'data/faithful.dat',
    delim = ' ',
    skip = 26,
    col_names = c('index', 'eruption_time', 'waiting_time')
)

In summary, we needed to:

• manually specify the delimiter as a single whitespace
• skip 26 lines of metadata
• manually supply the column names index, eruption_time, waiting_time

Check that the number of rows/columns looks right, the column types are correct (numeric vs. character), and there are no parsing warnings from read_delim().

📝 Note About Loading Data

• It’s important to do it carefully + check results after!
  • will help reduce bugs and speed up your analyses down the road
• Think of it as tying your shoes before you run 👟; not exciting, but if done wrong it will trip you up later!

📅 Upcoming Due Dates

• worksheet_reading due THIS SATURDAY 11:59pm
• tutorial_reading due THIS SATURDAY 11:59pm

There are NO late submissions/concessions for worksheets. We will drop the lowest worksheet and lowest tutorial score. See policy for late registrants.

❓ Questions?

If not, let’s work together on this week’s worksheet.

• Want to save these slides? Right click → Print → Save to PDF.
• Your code and notes will only save if you save as a PDF!