Class RMSNormLayer

java.lang.Object
smile.deep.layer.RMSNormLayer
All Implemented Interfaces:
Function<Tensor,Tensor>, Layer
public class RMSNormLayer extends Object implements Layer
Root Mean Square Layer Normalization. RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm.

  • Constructor Details

    • RMSNormLayer

      public RMSNormLayer(int dim)
      Constructor.
      Parameters:
      dim - the layer size.

    • RMSNormLayer

      public RMSNormLayer(int dim, double eps)
      Constructor.
      Parameters:
      dim - the layer size.
      eps - the term added to the denominator to improve numerical stability.

  • Method Details

    • forward

      public Tensor forward(Tensor input)
      Description copied from interface: Layer
      Forward propagation (or forward pass) through the layer.
      Specified by:
      forward in interface Layer
      Parameters:
      input - the input tensor.
      Returns:
      the output tensor.

    • asTorch

      public org.bytedeco.pytorch.Module asTorch()
      Description copied from interface: Layer
      Returns the PyTorch Module object.
      Specified by:
      asTorch in interface Layer
      Returns:
      the PyTorch Module object.