smile.nlp

Class SimpleCorpus

java.lang.Object

smile.nlp.SimpleCorpus

All Implemented Interfaces:

Corpus

public class SimpleCorpus
extends java.lang.Object
implements Corpus

An in-memory text corpus. Useful for text feature engineering.

Constructor Summary

Constructors

Constructor and Description
SimpleCorpus() Constructor.
SimpleCorpus(SentenceSplitter splitter, Tokenizer tokenizer, StopWords stopWords, Punctuations punctuations) Constructor.

Method Summary

Modifier and Type	Method and Description
Text	add(Text text) Add a document to the corpus.
int	getAverageDocumentSize() Returns the average size of documents in the corpus.
int	getBigramFrequency(Bigram bigram) Returns the total frequency of the bigram in the corpus.
java.util.Iterator<Bigram>	getBigrams() Returns an iterator over the bigrams in the corpus.
long	getNumBigrams() Returns the number of bigrams in the corpus.
int	getNumDocuments() Returns the number of documents in the corpus.
int	getNumTerms() Returns the number of unique terms in the corpus.
int	getTermFrequency(java.lang.String term) Returns the total frequency of the term in the corpus.
java.util.Iterator<java.lang.String>	getTerms() Returns an iterator over the terms in the corpus.
java.util.Iterator<Relevance>	search(RelevanceRanker ranker, java.lang.String term) Returns an iterator over the set of documents containing the given term in descending order of relevance.
java.util.Iterator<Relevance>	search(RelevanceRanker ranker, java.lang.String[] terms) Returns an iterator over the set of documents containing (at least one of) the given terms in descending order of relevance.
java.util.Iterator<Text>	search(java.lang.String term) Returns an iterator over the set of documents containing the given term.
long	size() Returns the number of words in the corpus.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleCorpus

public SimpleCorpus()

Constructor.

SimpleCorpus

public SimpleCorpus(SentenceSplitter splitter,
                    Tokenizer tokenizer,
                    StopWords stopWords,
                    Punctuations punctuations)

Constructor.

Parameters:

splitter - the sentence splitter.

tokenizer - the word tokenizer.

stopWords - the set of stop words to exclude.

punctuations - the set of punctuation marks to exclude. Set to null to keep all punctuation marks.

Method Detail

add

public Text add(Text text)

Add a document to the corpus.

size

public long size()

Description copied from interface: Corpus

Returns the number of words in the corpus.

Specified by:

size in interface Corpus

getNumDocuments

public int getNumDocuments()

Description copied from interface: Corpus

Returns the number of documents in the corpus.

Specified by:

getNumDocuments in interface Corpus

getNumTerms

public int getNumTerms()

Description copied from interface: Corpus

Returns the number of unique terms in the corpus.

Specified by:

getNumTerms in interface Corpus

getNumBigrams

public long getNumBigrams()

Description copied from interface: Corpus

Returns the number of bigrams in the corpus.

Specified by:

getNumBigrams in interface Corpus

getAverageDocumentSize

public int getAverageDocumentSize()

Description copied from interface: Corpus

Returns the average size of documents in the corpus.

Specified by:

getAverageDocumentSize in interface Corpus

getTermFrequency

public int getTermFrequency(java.lang.String term)

Description copied from interface: Corpus

Returns the total frequency of the term in the corpus.

Specified by:

getTermFrequency in interface Corpus

getBigramFrequency

public int getBigramFrequency(Bigram bigram)

Description copied from interface: Corpus

Returns the total frequency of the bigram in the corpus.

Specified by:

getBigramFrequency in interface Corpus

getTerms

public java.util.Iterator<java.lang.String> getTerms()

Description copied from interface: Corpus

Returns an iterator over the terms in the corpus.

Specified by:

getTerms in interface Corpus

getBigrams

public java.util.Iterator<Bigram> getBigrams()

Description copied from interface: Corpus

Returns an iterator over the bigrams in the corpus.

Specified by:

getBigrams in interface Corpus

search

public java.util.Iterator<Text> search(java.lang.String term)

Description copied from interface: Corpus

Returns an iterator over the set of documents containing the given term.

Specified by:

search in interface Corpus

search

public java.util.Iterator<Relevance> search(RelevanceRanker ranker,
                                            java.lang.String term)

Description copied from interface: Corpus

Returns an iterator over the set of documents containing the given term in descending order of relevance.

Specified by:

search in interface Corpus

search

public java.util.Iterator<Relevance> search(RelevanceRanker ranker,
                                            java.lang.String[] terms)

Description copied from interface: Corpus

Returns an iterator over the set of documents containing (at least one of) the given terms in descending order of relevance.

Specified by:

search in interface Corpus