Package smile.llm.llama

Class FeedForward

java.lang.Object

smile.llm.llama.FeedForward

public class FeedForward
extends Object

Feedforward layer in Transformer. It has two linear transformations and an intermediate SiLU activation function.

Constructor Summary

Constructors

Constructor	Description
FeedForward(int dim, int hiddenDim, int multipleOf, Double ffnDimMultiplier)	Constructor.

Method Summary

Modifier and Type	Method	Description
Tensor	forward(Tensor x)	Feed forward.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

FeedForward

public FeedForward(int dim,
 int hiddenDim,
 int multipleOf,
 Double ffnDimMultiplier)

Constructor.

Parameters:

dim - the dimension of input tensor.

hiddenDim - the dimension of hidden layer. First, hiddenDim is set to two-thirds of the provided hiddenDim value. If ffnDimMultiplier is provided, hiddenDim is further multiplied by this value. The hiddenDim is then adjusted to ensure it is a multiple of multipleOf.

multipleOf - make SwiGLU hidden layer size multiple of large power of 2.

ffnDimMultiplier - the multiplier for the hidden dimension of the feedforward layers.

Method Details

forward

public Tensor forward(Tensor x)

Feed forward.

Parameters:

x - the input tensor.

Returns:

the output tensor.