Package smile.llm

Class PositionalEncoding

java.lang.Object

smile.llm.PositionalEncoding

All Implemented Interfaces:

Function<Tensor,Tensor>, Layer

public class PositionalEncoding
extends Object
implements Layer

Positional encoding injects some information about the relative or absolute position of the tokens in the sequence. The positional encodings have the same dimension as the embeddings, so that the two can be summed. This class uses sine and cosine functions of different frequencies.

Constructor Summary

Constructors

Constructor	Description
PositionalEncoding(int dModel)	Constructor.
PositionalEncoding(int dModel, double dropout, int maxLen)	Constructor.

Method Summary

Modifier and Type	Method	Description
org.bytedeco.pytorch.Module	asTorch()	Returns the PyTorch Module object.
Tensor	forward(Tensor input)	Forward propagation (or forward pass) through the layer.
PositionalEncoding	to(Device device)	Moves the encoder to a device.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.function.Function

andThen, compose

Methods inherited from interface smile.deep.layer.Layer

apply, isTraining

Constructor Details

PositionalEncoding

public PositionalEncoding(int dModel)

Constructor.

Parameters:

dModel - the number of expected features in the token embedding.

PositionalEncoding

public PositionalEncoding(int dModel,
 double dropout,
 int maxLen)

Constructor.

Parameters:

dModel - the number of expected features in the token embedding.

dropout - the dropout probability.

maxLen - the maximum length of token sequence.

Method Details

forward

public Tensor forward(Tensor input)

Description copied from interface: Layer

Forward propagation (or forward pass) through the layer.

Specified by:

forward in interface Layer

Parameters:

input - the input tensor.

Returns:

the output tensor.

asTorch

public org.bytedeco.pytorch.Module asTorch()

Description copied from interface: Layer

Returns the PyTorch Module object.

Specified by:

asTorch in interface Layer

Returns:

the PyTorch Module object.

to

public PositionalEncoding to(Device device)

Moves the encoder to a device.

Parameters:

device - the compute device.

Returns:

this encoder.