Package smile.llm.llama

Class Tokenizer

java.lang.Object

smile.llm.tokenizer.Tiktoken

smile.llm.llama.Tokenizer

All Implemented Interfaces:

Tokenizer

public class Tokenizer
extends Tiktoken

Custom tokenizer for Llama 3 models.

Field Summary

Fields inherited from class smile.llm.tokenizer.Tiktoken

ranks, specialTokens

Constructor Summary

Constructors

Constructor	Description
Tokenizer(Map<Bytes,Integer> ranks)	Constructor with default BOS, EOS, and special tokens.
Tokenizer(Map<Bytes,Integer> ranks, String bos, String eos, String... specialTokens)	Constructor.

Method Summary

Modifier and Type	Method	Description
int[]	encodeDialog(Message... dialog)	Encodes the messages of a dialog.
int[]	encodeMessage(Message message)	Encodes a message.
static Tokenizer	of(String path)	Loads a llama3 tokenizer model.

Methods inherited from class smile.llm.tokenizer.Tiktoken

allowSpecialTokens, decode, encode, encode, isSpecialTokenAllowed, load, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Tokenizer

public Tokenizer(Map<Bytes,Integer> ranks)

Constructor with default BOS, EOS, and special tokens.

Parameters:

ranks - The token to rank map.

Tokenizer

public Tokenizer(Map<Bytes,Integer> ranks,
 String bos,
 String eos,
 String... specialTokens)

Constructor.

Parameters:

ranks - The token to id map.

bos - beginning of sequence token.

eos - end of sequence token.

specialTokens - Optional special tokens.

Method Details

encodeMessage

public int[] encodeMessage(Message message)

Encodes a message.

Parameters:

message - the message.

Returns:

the tokens.

encodeDialog

public int[] encodeDialog(Message... dialog)

Encodes the messages of a dialog.

Parameters:

dialog - the messages.

Returns:

the tokens.

of

public static Tokenizer of(String path)
                    throws IOException

Loads a llama3 tokenizer model.

Parameters:

path - The llama3 model file path.

Returns:

a llama3 tokenizer.

Throws:

IOException - if fail to load the model.