Working with Files

In the previous lesson, you set up your Python project on your own computer. Now it is time to start working with some actual data.

In a real project, you usually do not type all your data directly into the code. Most of the time, that data comes from a file. Maybe it is a file exported by another application, or something another program has generated. And once you have processed that data, you will often want to save the results to another file. That way, the results are still available even after your program finishes running. That is what this lesson is about.

The data

Here is the data you will be working with. It is a file called reviews.csv, inside a folder called data in your review-stats project. It contains the same five reviews you have been using throughout the course, but this time each review also has a product name and a rating:

product,rating,review
kettle,5,"Great kettle! Boils fast, and it is quiet."
kettle,4,"The lid is stiff, but it boils fast. Great value!"
toaster,5,"Toasts evenly. Easy to clean, and the crumb tray is great."
headphones,4,"Great sound, easy to pair, and the battery lasts all week!"
headphones,2,"Not great. The left side is quiet, and it is not easy to fix?"

To follow along, create the data folder in your project and save those six lines in it as reviews.csv. The project should also still have the review_stats package from the earlier lessons, with tokenise, count_words and top_words in text_tools.py.

This raises two questions. How do you read this file using Python? And once you have processed the reviews, how do you save the results so another program can use them later?

Start with something simple. Every file on your computer has a pathKey termThe folders and the file name that locate a file: Path("data") / "notes.txt".. A path tells you where the file is located. It includes the folders you need to go through, followed by the file's name.

In Python, one of the easiest ways to work with paths is through something called Path. It is part of the pathlib module.

Joining parts with a slash

Suppose you want to refer to a file called notes.txt inside a folder called data. You import Path, and then write Path("data") / "notes.txt":

from pathlib import Path
notes = Path("data") / "notes.txt"
print(notes)

data/notes.txt

Notice the forward slash between the two parts. You are not dividing anything here. Path uses the slash operator to join parts of a path together. And it handles the appropriate path separator for the operating system you are using. So on macOS or Linux, printing this path gives data/notes.txt. On Windows, the same code prints the path with backslashes, data\notes.txt.

Relative paths

There is something important about the path you just created. You only wrote data/notes.txt. You did not specify the complete location on your computer. That is because this is a relative path.

A relative path is interpreted from something called the current working directory. The working directory is the folder your program is currently working from. In your notebook, that is your project folder. So when you write data/notes.txt, Python looks for a folder called data inside your project, and then a file called notes.txt inside that folder.

Does the file exist?

You can check whether the file exists with the exists method:

notes.exists()

False

Right now you get False, because you have not created the file yet.

Making a folder

So create it. First, you need to make sure the data folder exists:

Path("data").mkdir(exist_ok=True)

The mkdir method creates a directory. And exist_ok=True means that if the folder already exists, Python will not raise an error just because of that. Your data folder is already there, holding reviews.csv, so this cell does nothing and shows nothing.

Now that you have your folder, you can create the file. One simple way to do that is with the write_text method. You call notes.write_text, pass in two lines of text, and specify encoding="utf-8". The text contains "Great kettle!" on the first line and "Too loud." on the second, and each line ends with a newline character, \n:

notes.write_text("Great kettle!\nToo loud.\n", encoding="utf-8")

24

When you run this, Python writes the text into the file. The 24 under the cell is the value write_text returns: the number of characters it wrote. Check again whether the file exists:

notes.exists()

True

This time you get True, so the file is there. And if you call read_text, Python reads the entire file and gives you its contents as one string:

notes.read_text(encoding="utf-8")

'Great kettle!\nToo loud.\n'

You get "Great kettle!", followed by a newline character, then "Too loud.", followed by another newline.

One line at a time

That is one way to read and write text files. But what if the file contains thousands of lines? You might not want to read the entire thing into one big string. Instead, you can open the file and process it one line at a time. For that, you use open.

You write with open(notes, encoding="utf-8") as f:. This opens your file and gives you a file objectKey termWhat open gives back; you read from it or write to it. with closes it., which you are calling f. Inside the with block, you can loop over that file object, and each iteration gives you the next line.

Removing the newline

What if you only want the text itself? You can use rstrip. For example, line.rstrip("\n") removes newline characters from the end of the string. If you collect those values into a list, you have your two lines without the newline characters:

with open(notes, encoding="utf-8") as f:
    lines = [line.rstrip("\n") for line in f]
lines

['Great kettle!', 'Too loud.']

Why with?

There is something else to notice. You used a with statement to open the file. Why?

Because the with statement takes care of closing the file when you are finished with it. As soon as Python leaves the with block, it closes the file automatically. You can see that on the file object, whose closed attribute is now True:

f.closed

True

Even if an error happens inside the block, Python still closes the file as part of leaving that context. So throughout this course, whenever you use open, you will normally use it with a with statement.

Now look at another important part of opening files: the mode. When you call open, you can specify what you want to do with the file.

So far, you have been reading files. The default mode is "r", which stands for read.

Append mode

But what if you want to add more text to an existing file? For that, you use mode "a". The "a" stands for append. Open your notes file in append mode, write another line, "Easy to clean.", and then read the file again, split into lines:

with open(notes, "a", encoding="utf-8") as f:
    f.write("Easy to clean.\n")
notes.read_text(encoding="utf-8").splitlines()

['Great kettle!', 'Too loud.', 'Easy to clean.']

Now you have three lines: "Great kettle!", "Too loud." and "Easy to clean.". Append mode adds new content to the end of the file without removing what is already there.

Write mode

A file that does not exist

What happens if you try reading a file that does not exist, for example data/missing.txt?

open("data/missing.txt", encoding="utf-8")

FileNotFoundError: [Errno 2] No such file or directory: 'data/missing.txt'

Python raises a FileNotFoundError. The error tells you exactly what went wrong, and which path it could not find.

There is one more thing you have probably noticed. Every time you have worked with text files, you have written encoding="utf-8". Why do you need to specify an encoding at all?

Here is the thing. A file does not actually store letters or words. It stores bytes. And an encodingKey termThe rule that turns text into bytes, and bytes back into text. This course always uses UTF-8. tells Python how to convert between those bytes and the characters you see on the screen.

Characters and bytes

Take a simple example: the word "café". It has four characters, so len("café") gives 4. Now encode that string using UTF-8 with the encode method, and measure the result too:

len("café"), len("café".encode("utf-8"))

(4, 5)

The letters "c", "a" and "f" each use one byte. But the "é", with the accent, uses two bytes in UTF-8. You can see them in the encoded bytes, written as \xc3 and \xa9:

"café".encode("utf-8")

b'caf\xc3\xa9'

So even though the string has four characters, its UTF-8 representation contains five bytes.

Reading with the wrong encoding

What happens if you write that text using UTF-8, but read it back using a different encoding? Write "café" to a file using UTF-8:

cafe = Path("data/cafe.txt")
cafe.write_text("café", encoding="utf-8")

4

Then read the same file using an older encoding called cp1252, which is associated with Windows:

cafe.read_text(encoding="cp1252")

'café'

Instead of "café", you get "café". That is because the two bytes representing "é" in UTF-8 are being interpreted as two separate characters.

So the encoding you use to read a file matters. If it does not match the encoding used to write the file, you might get incorrect text or an error. And if you do not specify an encoding, Python's default can depend on the environment and its configuration. That is why this course always uses UTF-8 explicitly. It makes your code more predictable across different computers.

When you read the file again using UTF-8, you get the correct text:

cafe.read_text(encoding="utf-8")

'café'

Now that you know how to work with text files, get back to the reviews. Remember the reviews.csv file from the beginning? It is inside your data folder, and it contains five reviews, along with their product names and ratings.

CSVKey termA table stored as text, one row per line, values separated by commas. A value with a comma in it is written in quotes. stands for comma-separated values. It is a very common format for storing tabular data: a table saved as plain text. Each line represents a row, and the values in that row are separated by commas. Usually, the first row contains the column names. In this file, there are columns for the product, the rating and the review text.

DictReader

Python has a built-in module specifically for working with CSV files. It is called csv. You import it, then open reviews.csv using with open. You specify encoding="utf-8" as usual, and you also add newline="". That second option is recommended when working with Python's csv module, because it lets the module handle newline characters correctly.

Inside the with block, you create a csv.DictReader and pass it your file object. A DictReader reads each row and gives you a dictionary. The keys come from the column names in the header, and the values come from that particular row. Convert the rows into a list so you can inspect them:

import csv
with open("data/reviews.csv", encoding="utf-8", newline="") as f:
    rows = list(csv.DictReader(f))
len(rows)

5

You have five rows. Look at the first one:

rows[0]

{'product': 'kettle', 'rating': '5', 'review': 'Great kettle! Boils fast, and it is quiet.'}

The product is "kettle", the rating is "5", and the review says "Great kettle! Boils fast, and it is quiet.".

Commas inside a value

Notice something. The review itself contains commas. So how does Python know those commas are part of the review, rather than separators between columns?

In the CSV file, that review is surrounded by quotation marks. The CSV reader understands that, so it treats the entire quoted review as one value. That is one reason you use the csv module instead of simply splitting each line on commas.

Adding two ratings

Converting the ratings

So how do you fix it? You convert the ratings to integers, using int on each rating. Then you can calculate the average: add the ratings together using sum, and divide by the number of ratings using len:

ratings = [int(row["rating"]) for row in rows]
print(ratings)
print(sum(ratings) / len(ratings))

[5, 4, 5, 4, 2]
4.0

Now you have a list of numbers, 5, 4, 5, 4 and 2, and an average rating of 4.0.

So remember: when you read data from a CSV file, do not assume numeric-looking values are already numbers. You will often need to convert them yourself.

CSV is great when you are working with tables. But what if the data you want to save is not a table? For example, your word counts. Remember, your count_words function gives you a dictionary, where each word is a key and its count is the value. You could convert that into a CSV file. But there is another format that is particularly useful for storing structured data like dictionaries and lists.

It is called JSONKey termA text format for dictionaries, lists, strings and numbers. It has no tuples., which stands for JavaScript Object Notation. Despite the name, it is used by all kinds of programming languages, not just JavaScript. It is a text-based format that can represent objects, arrays, strings, numbers, booleans and null values. In Python, you work with JSON using the built-in json module.

Saving with json.dump

Suppose you have a word-count dictionary, and you want to save it to a file. You open a file in write mode, and then call json.dump(counts, f, indent=2). Here, counts is the dictionary you want to save, and f is the file you are writing to.

The indent=2 option makes the JSON easier for a person to read. Instead of putting everything on one long line, it formats the data with indentation. So when you print the resulting file, you can see each word and its count clearly:

import json
counts = {"great": 5, "easy": 3, "boils": 2}
with open("data/counts.json", "w", encoding="utf-8") as f:
    json.dump(counts, f, indent=2)
print(Path("data/counts.json").read_text(encoding="utf-8"))

{
  "great": 5,
  "easy": 3,
  "boils": 2
}

Loading with json.load

What if you want to read that file back into Python? You open it in read mode, and call json.load(f):

with open("data/counts.json", encoding="utf-8") as f:
    back = json.load(f)
print(back == counts, type(back))

True <class 'dict'>

Python gives you a dictionary again, equal to the one you saved. So you have successfully saved your data to a file and loaded it back.

Now bring everything together. You started with a CSV file containing five product reviews. The goal was to read those reviews, process the text, and save the results.

You already have the functions you need. You created them in the previous modules. So import tokenise, count_words and top_words from your review_stats package.

You already read the rows from reviews.csv into rows. For each row, you take the review text and pass it to tokenise. That gives you the words from that review, and you collect those words into one list. Then you pass that list to count_words, leaving out your stop words. Now you have the word counts for all five reviews. Finally, look at the five most common words:

from review_stats.text_tools import count_words, tokenise, top_words
STOP_WORDS = frozenset({"the", "and", "it", "is", "a", "to", "but", "all", "not"})
tokens = []
for row in rows:
    tokens += tokenise(row["review"])
counts = count_words(tokens, STOP_WORDS)
top_words(counts)

[('great', 5), ('easy', 3), ('boils', 2), ('fast', 2), ('quiet', 2)]

You get "great" with a count of five, "easy" with a count of three, and "boils", "fast" and "quiet", each with a count of two.

Does that result look familiar? It should. It is the same result you got in the previous module, when you typed the reviews directly into your notebook. The difference is that this time, your data came from a file. So you can process new reviews simply by changing the input data, without rewriting the logic.

Saving the counts

Now save those counts. First, create a folder called results, using mkdir with exist_ok=True so it will not complain if the folder already exists. Then build a path for counts.json inside that folder, and use json.dump to write your counts dictionary into the file:

Path("results").mkdir(exist_ok=True)
with open(Path("results") / "counts.json", "w", encoding="utf-8") as f:
    json.dump(counts, f, indent=2)
len(counts)

22

Now you have a JSON file containing the word counts. The dictionary contains 22 different words. And that file will still be there after your program finishes. So another program can read it later, or you can open it yourself and inspect the results.

Go back to the two questions you started with. How does a Python program read a file it did not create? And how does it save results that are still available after the program finishes?

• You use paths to tell Python where files are located. A relative path is interpreted from the current working directory.
• You use open to read and write files, and the with statement makes sure those files get closed when you are done.
• File modes say what you want to do with a file. "r" reads, "a" appends, and "w" replaces the contents. Be careful with "w": it clears the existing contents.
• Files store bytes, and an encoding tells Python how to turn those bytes into text and back. That is why this course uses UTF-8 explicitly.
• CSV is useful for tables, and csv.DictReader lets you read each row as a dictionary. The values come in as strings, so you need to convert them before doing numeric calculations.
• JSON is useful for saving structured data like dictionaries and lists. You use json.dump to write it, and json.load to read it back.

So now your project can read data from files, process it, and save the results. That is a big step toward building programs that work with real data.

Your challenge

You have already saved all your word counts into counts.json. This time, save just the five most common words into a CSV file.

1. Call it top_words.csv, and put it inside the results folder.
2. The file should have two columns, word and count. Do not forget the header row.
3. Once you have written the file, read it back using csv.DictReader.
4. Check that the first row contains "great", with a count of 5.

What comes next

There is one more problem you have not dealt with yet. So far, the CSV file has been pretty clean. All five reviews have valid ratings, and every rating is a whole number.

But what if a sixth review comes in, and instead of the number 5, someone has written the word "five"? When you try to convert that using int, Python raises a ValueError. And if you do not handle that error, your program stops right there. It does not finish processing the reviews, and it does not calculate the average.

So how do you deal with bad data without letting one incorrect value stop the entire program? In the next lesson, you will learn how to handle errors, skip rows you cannot use, and keep the rest of your program running.