Package smile.nlp

Returns the bag of words. The bag-of-words model is a simple representation of text as the bag of its words, disregarding grammar and word order but keeping multiplicity.

Returns the binary bag of words. Presence/absence is used instead of frequencies.

Identify bigram collocations whose p-value is less than the given threshold.

Identify bigram collocations (words that often appear consecutively) within corpora. They may also be used to find other associations between word occurrences.

Creates an in-memory text corpus.

Returns the document frequencies, i.e. the number of documents that contain term.

Keyword extraction from a single document using word co-occurrence statistical information.

The Paice/Husk Lancaster stemming algorithm. The stemmer is a conflation based iterative stemmer. The stemmer, although remaining efficient and easily implemented, is known to be very strong and aggressive. The stemmer utilizes a single table of rules, each of which may specify the removal or replacement of an ending.

An Apiori-like algorithm to extract n-gram phrases.

Normalizes Unicode text.

Porter's stemming algorithm. The stemmer is based on the idea that the suffixes in the English language are mostly made up of a combination of smaller and simpler suffixes. This is a linear step stemmer. Specifically it has five steps applying rules within each step. Within each step, if a suffix rule matched to a word, then the conditions attached to that rule are tested on what would be the resulting stem, if that suffix was removed, in the way defined by the rule. Once a Rule passes its conditions and is accepted the rule fires and the suffix is removed and control moves to the next step. If the rule is not accepted then the next rule in the step is tested, until either a rule from that step fires and control passes to the next step or there are no more rules in that step whence control moves to the next step.

Returns the (word, part-of-speech) pairs. The text should be a single sentence.

Part-of-speech taggers.

Splits English text into sentences. Given an English text, it returns a list of strings, where each element is an English sentence. By default, it treats occurrences of '.', '?' and '!' as sentence delimiters, but does its best to determine when an occurrence of '.' does not have this role (e.g. in abbreviations, URLs, numbers, etc.).

Converts a corpus to TF-IDF feature vectors, which are normalized to L2 norm 1.

Converts a bag of words to a feature vector by TF-IDF, which is normalized to L2 norm 1.

TF-IDF relevance score between a term and a document based on a corpus.

Converts a bag of words to a feature vector.

Converts a binary bag of words to a sparse feature vector.

Tokenizes English sentences with some differences from TreebankWordTokenizer, notably on handling not-contractions. If a period serves as both the end of sentence and a part of abbreviation, e.g. etc. at the end of sentence, it will generate tokens of "etc." and "." while TreebankWordTokenizer will generate "etc" and ".".