Strings

In the previous lesson you learned about dictionaries, and saw how a dictionary can help you count things. Towards the end, it talked about counting the words in product reviews. But there is a problem to solve first. A product review is not already a list of words. It is just one long string, and that string might contain capital letters, punctuation and extra spaces.

Here is an example. Imagine someone has left a review for a kettle:

review = '  Great kettle! Boils fast, and it is QUIET.  '
review

'  Great kettle! Boils fast, and it is QUIET.  '

The review reads "Great kettle! Boils fast, and it is QUIET." But there are also two extra spaces at the beginning and two at the end. They are part of the string.

Why the text needs cleaning

If you want to count the words in this review, you cannot simply use the text as it is. If one customer writes Great with a capital G and another writes great in lowercase, Python treats those as different strings. And if a word has punctuation attached to it, like quiet. with a full stop, that is also different from quiet without the full stop.

So before you can count words properly, you need to clean up the text, and that is what this lesson teaches. By the end of it, you will take that messy review and turn it into a list of eight clean words: all lowercase, no punctuation and no extra spaces. And you will do it with just a few lines of Python.

The question

There is a catch. Strings are immutable: you cannot change the characters inside a string. So how can you clean and transform a string at all? This lesson finds out.

First, you need to understand what a string actually is. A stringKey termAn immutable sequence of characters, indexed and sliced like a list. is an ordered sequence of characters. Because it is a sequence, many of the things you learned about lists and tuples also work with strings.

Indexing

Take indexing, for example. Create a variable called word and store the string "kettle" in it. Then ask for the character at index 0:

word = "kettle"
word[0]

'k'

Python gives you the character k, the first character. Index -1 gives the last one:

word[-1]

'e'

That is exactly like negative indexing with lists and tuples. And len works too:

len(word)

6

You get 6, because kettle contains six characters.

Slicing

Just like lists, strings also support slicing:

word[1:4]

'ett'

The slice starts at index 1 and stops just before index 4. So you get the characters at indices 1, 2 and 3, which are e, t and t.

Now back to the question from the start of this lesson. If strings are immutable, how do you change them? First, see what happens when you try to modify one directly.

Trying to change a character

Building a new string

That does not mean you cannot create a different string. Suppose you want to capitalize the first letter of kettle:

"K" + word[1:]

'Kettle'

This takes an uppercase letter K and combines it with everything from index 1 onward in word. The result is Kettle with a capital K.

Notice that you did not modify the original string. You created a new one. That is the basic idea behind string transformations in Python.

String methods: lower

Strings also have methods, just like lists and dictionaries, and you call them with a dot after the string or variable name. One of the most useful is lower().

Remember the product review? It contained words like Great and Boils with capital letters, and QUIET written entirely in uppercase. Call lower() on it:

review.lower()

'  great kettle! boils fast, and it is quiet.  '

Python returns a lowercase version of the review. Great becomes great with a lowercase g, Boils becomes boils with a lowercase b, and QUIET becomes quiet, all in lowercase.

Does lower change the review?

Keeping the result

So if you want to keep the lowercase version, you need to assign the result:

review = review.lower()
review

'  great kettle! boils fast, and it is quiet.  '

Now the name review refers to the lowercase string. The original string object was not modified. You simply rebound the variable name to the result. That is the same distinction between mutation and reassignment you learned earlier.

Now deal with those two extra spaces at the beginning of the review and two at the end. Start again from the original review:

review = '  Great kettle! Boils fast, and it is QUIET.  '
review.strip()

'Great kettle! Boils fast, and it is QUIET.'

The strip() method removes leading and trailing whitespaceKey termSpacing characters such as spaces, tabs and newlines. from a string. Whitespace includes ordinary spaces, tabs and newline characters. Python returns the review without the extra spaces at the beginning and at the end. But it does not remove the spaces between the words. Those stay exactly where they are. So strip() is useful when you want to clean up unwanted spaces around some text.

Method chaining

Because strip() returns a string, you can immediately call another string method on that result:

review.strip().lower()

'great kettle! boils fast, and it is quiet.'

First, Python calls strip(), which returns a string without the extra whitespace at the ends. Then Python calls lower() on that result, and it returns the lowercase version of the complete string.

This is called method chaining: calling one method after another, each one using the result of the previous method.

replace

Now for the punctuation. The review still contains an exclamation mark, a comma and a full stop, and you do not want those characters attached to the words when you count them. For that, you can use replace().

The replace() method takes two arguments: the text you want to find, and the text you want to put in its place. Suppose you have a string with three exclamation marks in it, and you want to remove them:

text = "great!! quiet!"
text.replace("!", "")

'great quiet'

The second argument is an empty string, so you are telling Python to replace each exclamation mark with nothing. The result is clean text, and all three exclamation marks are gone.

Just like lower() and strip(), replace() returns a string without modifying the original. This is useful because you can remove different punctuation marks one by one, and you will do exactly that when you build the word-cleaning program.

split

There is one more important method you need: split(). Say you have this string, and you want to turn it into individual words:

text = "boils fast and quiet"
text.split()

['boils', 'fast', 'and', 'quiet']

text.split() returns a list, not a string. Here it is a list containing four strings: 'boils', 'fast', 'and' and 'quiet'. So split() takes one string and separates it into smaller strings. By default, it splits on whitespace. This is exactly what the word-counting project needs.

join

What if you want to go in the opposite direction? Suppose you already have a list of words, and you want to combine them into one string. For this, Python has a method called join().

The syntax is a little different from what you might expect. You call join() on the separator, not on the list. For example, take a list containing "great" and "kettle":

words = ["great", "kettle"]
" ".join(words)

'great kettle'

The string before .join is a single space, so you are telling Python to put that space between the elements. The result is one complete string. So join() combines strings using whatever separator you provide.

Now you have everything you need to solve the problem this lesson started with. The messy kettle review contains capital letters, punctuation and extra spaces, and you want to turn it into a list of clean words. Time to build a simple tokenizer.

Strip and lowercase

First, remove the extra whitespace at the beginning and end, and convert everything to lowercase. You can do both with method chaining:

text = review.strip().lower()
text

'great kettle! boils fast, and it is quiet.'

Now text contains the lowercase review without extra spaces around it. But the punctuation is still there.

Remove the punctuation in a loop

For this example, you will handle four punctuation characters: a full stop, a comma, an exclamation mark and a question mark. Put those four characters in one string, ".,!?", and loop through them one at a time. Strings are also iterable, so a for loop can go through their characters.

On each iteration, call replace() and replace the current punctuation character with an empty string:

for mark in ".,!?":
    text = text.replace(mark, "")
text

'great kettle boils fast and it is quiet'

Here is what happens. On the first iteration, mark is a full stop, so Python removes all full stops from text. On the second iteration, mark is a comma, so it removes all commas. Then it removes exclamation marks, and finally question marks.

Notice that the loop keeps assigning the result back to text. That is because replace() does not modify a string in place. It returns a new string, and you need to keep that result for the next iteration.

Split into words

With the punctuation gone, you are ready to split the text into words:

words = text.split()
words

['great', 'kettle', 'boils', 'fast', 'and', 'it', 'is', 'quiet']

That is it. You get a list of eight strings: great, kettle, boils, fast, and, it, is and quiet. All lowercase, no punctuation and no extra spaces. Exactly what you wanted.

Tokens

What you have just built is a very simple tokenizer. A tokenizer is a program that breaks text into smaller units called tokens. In this example, each token is a word, so the list contains eight tokens:

len(words)

8

This lesson started with a messy product review and the goal of turning it into a list of clean words.

First, you learned that strings are ordered sequences of characters, so you can index and slice them, just like lists and tuples. You also learned that strings are immutable, which means methods do not change the original string in place. Instead, they return a result, and if you want to keep that result, you need to store it.

Then you explored several useful string methods: lower() for converting text to lowercase, strip() for removing whitespace from both ends, replace() for swapping one piece of text for another, split() for breaking a string into a list, and join() for combining a list of strings into one. Finally, you used these methods to build a simple tokenizer, and now you have a list you can use in the word-counting project.

Your challenge

Here is another product review:

review = "  Loud. Very loud! Returned it, sadly.  "

It has capital letters, punctuation and spaces at both ends. Run this review through the tokenizer you just built, then answer three questions:

1. How many tokens do you get?
2. How many times does the word loud appear?
3. Can you join those tokens back together into one clean sentence, with exactly one space between each word?

Try solving it using the string methods from this lesson.

In the next lesson, you will learn a shorter way to build a new list from an existing one. It is called a comprehension.