BreakIteratorTokenizer

java.lang.Object

smile.nlp.tokenizer.BreakIteratorTokenizer

All Implemented Interfaces:: Function<String,String[]>, Tokenizer

public class BreakIteratorTokenizer extends Object implements Tokenizer

A word tokenizer based on the java.text.BreakIterator, which supports multiple natural languages (selected by locale setting).

Constructor Summary

Constructors

Constructor

Description

BreakIteratorTokenizer()

Constructor for the default locale.

BreakIteratorTokenizer(Locale locale)

Constructor for the given locale.
Method Summary

Modifier and Type

Method

Description

String[]

split(String text)

Splits the string into a list of tokens.

Methods inherited from class Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface Function
andThen, compose

Methods inherited from interface Tokenizer
apply

Constructor Details
- BreakIteratorTokenizer
  
  public BreakIteratorTokenizer()
  
  Constructor for the default locale.
- BreakIteratorTokenizer
  
  public BreakIteratorTokenizer(Locale locale)
  
  Constructor for the given locale.
  
  Parameters:
  
  locale - the locale.
Method Details
- split
  
  public String[] split(String text)
  
  Description copied from interface: Tokenizer
  
  Splits the string into a list of tokens.
  
  Specified by:
  
  split in interface Tokenizer
  
  Parameters:
  
  text - the text.
  
  Returns:
  
  the tokens.