Original article was published on Artificial Intelligence on Medium
Almost every Natural Language Processing (NLP) task requires text to be preprocessed before training a model. Deep learning models cannot use raw text directly, so it is up to us researchers to clean the text ourselves. Depending on the nature of the task, the preprocessing methods can be different. This tutorial will teach the most common preprocessing approach that can fit in with various NLP tasks using NLTK (Natural Language Toolkit).
- Popularity: NLTK is one of the leading platforms for dealing with language data.
- Simplicity: Provides easy-to-use APIs for a wide variety of text preprocessing methods
- Community: It has a large and active community that supports the library and improves it
- Open Source: Free and open-source available for Windows, Mac OSX, and Linux.
Now you know the benefits of NLTK, let’s get started!
- Removing Punctuation
- Stopword Filtering
- Part-of-Speech Tagger
All code displayed in this tutorial can be accessed in my Github repo.
Before preprocessing, we need to first download the NLTK library.
pip install nltk
Then, we can import the library in our Python notebook and download its contents.
As an example, we grab the first sentence from the book Pride and Prejudice as the text. We convert the sentence to lowercase via
To remove punctuation, we save only the characters that are not punctuation, which can be checked by using
Strings can be tokenized into tokens via
We can use
nltk.corpus.stopwords.words(‘english’) to fetch a list of stopwords in the English dictionary. Then, we remove the tokens that are stopwords.
We stem the tokens using
nltk.stem.porter.PorterStemmer to get the stemmed tokens.
Lastly, we can use
nltk.pos_tag to retrieve the part of speech of each token in a list.
The full notebook can be seen here.
Combining all Together
We can combine all the preprocessing methods above and create a
preprocess function that takes in a .txt file and handles all the preprocessing. We print out the tokens, filtered words (after stopword filtering), stemmed words, and POS, one of which is usually passed on to the model or for further processing. We use the Pride and Prejudice book (accessible here) and preprocess it.
This notebook can be accessed here.
Text preprocessing is an important first step for any NLP application. In this tutorial, we discussed several popular preprocessing approaches using NLTK: lowercase, removing punctuation, tokenization, stopword filtering, stemming, and part-of-speech tagger.