Package smile.llm

Class PositionalEncoding

java.lang.Object

smile.llm.PositionalEncoding

All Implemented Interfaces:

Function<Tensor,Tensor>, Layer

public class PositionalEncoding
extends Object
implements Layer

Positional encoding in original Transformer. Positional encoding injects some information about the absolute position of the tokens in the sequence. The positional encodings have the same dimension as the embeddings, so that the two can be summed. This class uses sine and cosine functions of different frequencies.

Constructor Summary

Constructors

Constructor	Description
PositionalEncoding(int dim, int end)	Constructor.
PositionalEncoding(int dim, int end, double theta)	Constructor.

Method Summary

Modifier and Type	Method	Description
org.bytedeco.pytorch.Module	asTorch()	Returns the PyTorch Module object.
Tensor	forward(Tensor input)	Forward propagation (or forward pass) through the layer.
PositionalEncoding	to(Device device)	Moves the encoder to a device.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.function.Function

andThen, compose

Methods inherited from interface smile.deep.layer.Layer

apply, to

Constructor Details

PositionalEncoding

public PositionalEncoding(int dim,
 int end)

Constructor.

Parameters:

dim - the dimension of the frequency tensor.

end - the end index for precomputing frequencies.

PositionalEncoding

public PositionalEncoding(int dim,
 int end,
 double theta)

Constructor.

Parameters:

dim - the dimension of the frequency tensor.

end - the end index for precomputing frequencies.

theta - the scaling factor for frequency computation.

Method Details

forward

public Tensor forward(Tensor input)

Description copied from interface: Layer

Forward propagation (or forward pass) through the layer.

Specified by:

forward in interface Layer

Parameters:

input - the input tensor.

Returns:

the output tensor.

asTorch

public org.bytedeco.pytorch.Module asTorch()

Description copied from interface: Layer

Returns the PyTorch Module object.

Specified by:

asTorch in interface Layer

Returns:

the PyTorch Module object.

to

public PositionalEncoding to(Device device)

Moves the encoder to a device.

Specified by:

to in interface Layer

Parameters:

device - the compute device.

Returns:

this encoder.