In our last post, we went over a range of options to perform approximate sentence matching in Python, an import task for many natural language processing and machine learning tasks. To begin, we defined terms like: tokens: a word, number, or other "discrete" unit of text. stems: words that have had their "inflected" pieces removed based on
Let's imagine you have a sentence of interest. You'd like to find all occurrences of this sentence within a corpus of text. How would you go about this? The most obvious answer is to look for exact matches of the sentence. You'd search through every sentence of your corpus, checking to see if every character of the
Back in 2011, I covered the relative performance difference of the most popular libraries for text processing in R and Python. In case you can't guess the answer, Python and NLTK won by a significant margin over R and tm. Text processing with R seemed simple on paper, but performance and flexibility limitations have
