Package smile.deep.layer

Class RMSNormLayer

java.lang.Object

smile.deep.layer.RMSNormLayer

All Implemented Interfaces:

Function<Tensor,Tensor>, Layer

public class RMSNormLayer
extends Object
implements Layer

Root Mean Square Layer Normalization. RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm.

Constructor Summary

Constructors

Constructor	Description
RMSNormLayer(int dim)	Constructor.
RMSNormLayer(int dim, double eps)	Constructor.

Method Summary

Modifier and Type	Method	Description
org.bytedeco.pytorch.Module	asTorch()	Returns the PyTorch Module object.
Tensor	forward(Tensor input)	Forward propagation (or forward pass) through the layer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.function.Function

andThen, compose

Methods inherited from interface smile.deep.layer.Layer

apply, to, to

Constructor Details

RMSNormLayer

public RMSNormLayer(int dim)

Constructor.

Parameters:

dim - the layer size.

RMSNormLayer

public RMSNormLayer(int dim,
 double eps)

Constructor.

Parameters:

dim - the layer size.

eps - the term added to the denominator to improve numerical stability.

Method Details

forward

public Tensor forward(Tensor input)

Description copied from interface: Layer

Forward propagation (or forward pass) through the layer.

Specified by:

forward in interface Layer

Parameters:

input - the input tensor.

Returns:

the output tensor.

asTorch

public org.bytedeco.pytorch.Module asTorch()

Description copied from interface: Layer

Returns the PyTorch Module object.

Specified by:

asTorch in interface Layer

Returns:

the PyTorch Module object.