Class Layer

java.lang.Object
smile.base.mlp.Layer
All Implemented Interfaces:
Serializable, AutoCloseable
Direct Known Subclasses:
HiddenLayer, InputLayer, OutputLayer
public abstract class Layer extends Object implements AutoCloseable, Serializable
A layer in the neural network.
See Also:

  • Field Details

    • n

      protected final int n
      The number of neurons in this layer

    • p

      protected final int p
      The number of input variables.

    • dropout

      protected final double dropout
      The dropout rate. Dropout randomly sets input units to 0 with this rate at each step during training time, which helps prevent overfitting.

    • weight

      protected DenseMatrix weight
      The affine transformation matrix.

    • bias

      protected Vector bias
      The bias.

    • output

      protected transient ThreadLocal<Vector> output
      The output vector.

    • outputGradient

      protected transient ThreadLocal<Vector> outputGradient
      The output gradient.

    • weightGradient

      protected transient ThreadLocal<DenseMatrix> weightGradient
      The weight gradient.

    • biasGradient

      protected transient ThreadLocal<Vector> biasGradient
      The bias gradient.

    • weightGradientMoment1

      protected transient ThreadLocal<DenseMatrix> weightGradientMoment1
      The first moment of weight gradient.

    • weightGradientMoment2

      protected transient ThreadLocal<DenseMatrix> weightGradientMoment2
      The second moment of weight gradient.

    • biasGradientMoment1

      protected transient ThreadLocal<Vector> biasGradientMoment1
      The first moment of bias gradient.

    • biasGradientMoment2

      protected transient ThreadLocal<Vector> biasGradientMoment2
      The second moment of bias gradient.

    • weightUpdate

      protected transient ThreadLocal<DenseMatrix> weightUpdate
      The weight update.

    • biasUpdate

      protected transient ThreadLocal<Vector> biasUpdate
      The bias update.

    • mask

      protected transient ThreadLocal<byte[]> mask
      The dropout mask.

  • Constructor Details

    • Layer

      public Layer(int n, int p)
      Constructor. Randomly initialized weights and zero bias.
      Parameters:
      n - the number of neurons.
      p - the number of input variables (not including bias value).

    • Layer

      public Layer(int n, int p, double dropout)
      Constructor. Randomly initialized weights and zero bias.
      Parameters:
      n - the number of neurons.
      p - the number of input variables (not including bias value).
      dropout - the dropout rate.

    • Layer

      public Layer(DenseMatrix weight, Vector bias)
      Constructor.
      Parameters:
      weight - the weight matrix.
      bias - the bias vector.

    • Layer

      public Layer(DenseMatrix weight, Vector bias, double dropout)
      Constructor.
      Parameters:
      weight - the weight matrix.
      bias - the bias vector.
      dropout - the dropout rate.

  • Method Details

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable

    • getOutputSize

      public int getOutputSize()
      Returns the dimension of output vector.
      Returns:
      the dimension of output vector.

    • getInputSize

      public int getInputSize()
      Returns the dimension of input vector (not including bias value).
      Returns:
      the dimension of input vector.

    • weight

      public DenseMatrix weight()
      Returns the weight matrix.
      Returns:
      the weight matrix.

    • bias

      public Vector bias()
      Returns the bias vector.
      Returns:
      the bias vector.

    • output

      public Vector output()
      Returns the output vector.
      Returns:
      the output vector.

    • gradient

      public Vector gradient()
      Returns the output gradient vector.
      Returns:
      the output gradient vector.

    • propagate

      public void propagate(Vector x)
      Propagates the signals from a lower layer to this layer.
      Parameters:
      x - the lower layer signals.

    • propagateDropout

      public void propagateDropout()
      Propagates the output signals through the implicit dropout layer. Dropout randomly sets output units to 0. It should only be applied during training.

    • transform

      public abstract void transform(Vector x)
      The activation or output function.
      Parameters:
      x - the input and output values.

    • backpropagate

      public abstract void backpropagate(Vector lowerLayerGradient)
      Propagates the errors back to a lower layer.
      Parameters:
      lowerLayerGradient - the gradient vector of lower layer.

    • backpopagateDropout

      public void backpopagateDropout()
      Propagates the errors back through the (implicit) dropout layer.

    • computeGradientUpdate

      public void computeGradientUpdate(Vector x, double learningRate, double momentum, double decay)
      Computes the parameter gradient and update the weights.
      Parameters:
      x - the input vector.
      learningRate - the learning rate.
      momentum - the momentum factor.
      decay - weight decay factor.

    • computeGradient

      public void computeGradient(Vector x)
      Computes the parameter gradient for a sample of (mini-)batch.
      Parameters:
      x - the input vector.

    • update

      public void update(int m, double learningRate, double momentum, double decay, double rho, double epsilon)
      Adjust network weights by back-propagation algorithm.
      Parameters:
      m - the size of mini-batch.
      learningRate - the learning rate.
      momentum - the momentum factor.
      decay - weight decay factor.
      rho - RMSProp discounting factor for the history/coming gradient.
      epsilon - a small constant for numerical stability.

    • builder

      public static HiddenLayerBuilder builder(String activation, int neurons, double dropout, double param)
      Returns a hidden layer.
      Parameters:
      activation - the activation function.
      neurons - the number of neurons.
      dropout - the dropout rate.
      param - the optional activation function parameter.
      Returns:
      the layer builder.

    • input

      public static LayerBuilder input(int neurons)
      Returns an input layer.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • input

      public static LayerBuilder input(int neurons, double dropout)
      Returns an input layer.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • linear

      public static HiddenLayerBuilder linear(int neurons)
      Returns a hidden layer with linear activation function.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • linear

      public static HiddenLayerBuilder linear(int neurons, double dropout)
      Returns a hidden layer with linear activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • rectifier

      public static HiddenLayerBuilder rectifier(int neurons)
      Returns a hidden layer with rectified linear activation function.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • rectifier

      public static HiddenLayerBuilder rectifier(int neurons, double dropout)
      Returns a hidden layer with rectified linear activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • leaky

      public static HiddenLayerBuilder leaky(int neurons)
      Returns a hidden layer with leaky rectified linear activation function.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • leaky

      public static HiddenLayerBuilder leaky(int neurons, double dropout)
      Returns a hidden layer with leaky rectified linear activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • leaky

      public static HiddenLayerBuilder leaky(int neurons, double dropout, double a)
      Returns a hidden layer with leaky rectified linear activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      a - the parameter of leaky ReLU.
      Returns:
      the layer builder.

    • sigmoid

      public static HiddenLayerBuilder sigmoid(int neurons)
      Returns a hidden layer with sigmoid activation function.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • sigmoid

      public static HiddenLayerBuilder sigmoid(int neurons, double dropout)
      Returns a hidden layer with sigmoid activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • tanh

      public static HiddenLayerBuilder tanh(int neurons)
      Returns a hidden layer with hyperbolic tangent activation function.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • tanh

      public static HiddenLayerBuilder tanh(int neurons, double dropout)
      Returns a hidden layer with hyperbolic tangent activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • mse

      public static OutputLayerBuilder mse(int neurons, OutputFunction output)
      Returns an output layer with mean squared error cost function.
      Parameters:
      neurons - the number of neurons.
      output - the output function.
      Returns:
      the layer builder.

    • mle

      public static OutputLayerBuilder mle(int neurons, OutputFunction output)
      Returns an output layer with (log-)likelihood cost function.
      Parameters:
      neurons - the number of neurons.
      output - the output function.
      Returns:
      the layer builder.

    • of

      public static LayerBuilder[] of(int k, int p, String spec)
      Returns the layer builders given a string representation such as "Input(10, 0.2)|ReLU(50, 0.5)|Sigmoid(30, 0.5)|...".
      Parameters:
      k - the number of classes. k < 2 for regression.
      p - the number of input variables (not including bias value).
      spec - the hidden layer specification.
      Returns:
      the layer builders.