Package smile.base.mlp

Class MultilayerPerceptron

java.lang.Object

smile.base.mlp.MultilayerPerceptron

All Implemented Interfaces:

Serializable, AutoCloseable

Direct Known Subclasses:

MLP, MLP

public abstract class MultilayerPerceptron
extends Object
implements AutoCloseable, Serializable

Fully connected multilayer perceptron neural network. An MLP consists of at least three layers of nodes: an input layer, a hidden layer and an output layer. The nodes are interconnected through weighted acyclic arcs from each preceding layer to the following, without lateral or feedback connections. Each node calculates a transformed weighted linear combination of its inputs (output activations from the preceding layer), with one of the weights acting as a trainable bias connected to a constant input. The transformation, called activation function, is a bounded non-decreasing (non-linear) function.

See Also:

• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
protected double	clipNorm	The gradient clipping norm.
protected double	clipValue	The gradient clipping value.
protected double	epsilon	A small constant for numerical stability in RMSProp.
protected double	lambda	The L2 regularization factor, which is also the weight decay factor.
protected TimeFunction	learningRate	The learning rate.
protected TimeFunction	momentum	The momentum factor.
protected Layer[]	net	The input and hidden layers.
protected OutputLayer	output	The output layer.
protected int	p	The dimensionality of input data.
protected double	rho	The discounting factor for the history/coming gradient in RMSProp.
protected int	t	The training iterations.
protected ThreadLocal<double[]>	target	The buffer to store desired target value of training instance.

Constructor Summary

Constructors

Constructor	Description
MultilayerPerceptron(Layer... net)	Constructor.

Method Summary

Modifier and Type	Method	Description
protected void	backpropagate(boolean update)	Propagates the errors back through the network.
void	close()
double	getClipNorm()	Returns the gradient clipping norm.
double	getClipValue()	Returns the gradient clipping value.
double	getLearningRate()	Returns the learning rate.
double	getMomentum()	Returns the momentum factor.
double	getWeightDecay()	Returns the weight decay factor.
protected void	propagate(double[] x, boolean training)	Propagates the signals through the neural network.
void	setClipNorm(double clipNorm)	Sets the gradient clipping norm.
void	setClipValue(double clipValue)	Sets the gradient clipping value.
void	setLearningRate(TimeFunction rate)	Sets the learning rate.
void	setMomentum(TimeFunction momentum)	Sets the momentum factor.
void	setParameters(Properties params)	Sets MLP hyperparameters such as learning rate, weight decay, momentum, RMSProp, etc.
void	setRMSProp(double rho, double epsilon)	Sets RMSProp parameters.
void	setWeightDecay(double lambda)	Sets the weight decay factor.
String	toString()
protected void	update(int m)	Updates the weights for mini-batch training.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

p

protected int p

The dimensionality of input data.

output

protected OutputLayer output

The output layer.

net

protected Layer[] net

The input and hidden layers.

target

protected transient ThreadLocal<double[]> target

The buffer to store desired target value of training instance.

learningRate

protected TimeFunction learningRate

The learning rate.

momentum

protected TimeFunction momentum

The momentum factor.

rho

protected double rho

The discounting factor for the history/coming gradient in RMSProp.

epsilon

protected double epsilon

A small constant for numerical stability in RMSProp.

lambda

protected double lambda

The L2 regularization factor, which is also the weight decay factor.

clipValue

protected double clipValue

The gradient clipping value.

clipNorm

protected double clipNorm

The gradient clipping norm.

t

protected int t

The training iterations.

Constructor Details

MultilayerPerceptron

public MultilayerPerceptron(Layer... net)

Constructor.

Parameters:

net - the input layer, hidden layers, and output layer in order.

Method Details

close

public void close()

Specified by:

close in interface AutoCloseable

toString

public String toString()

Overrides:

toString in class Object

setLearningRate

public void setLearningRate(TimeFunction rate)

Sets the learning rate.

Parameters:

rate - the learning rate.

setMomentum

public void setMomentum(TimeFunction momentum)

Sets the momentum factor. momentum = 0.0 means no momentum.

Parameters:

momentum - the momentum factor.

setRMSProp

public void setRMSProp(double rho,
 double epsilon)

Sets RMSProp parameters.

Parameters:

rho - The discounting factor for the history/coming gradient.

epsilon - A small constant for numerical stability.

setWeightDecay

public void setWeightDecay(double lambda)

Sets the weight decay factor. After each weight update, every weight is simply "decayed" or shrunk according to w = w * (1 - 2 * eta * lambda).

Parameters:

lambda - the weight decay factor.

setClipValue

public void setClipValue(double clipValue)

Sets the gradient clipping value. If clip value is set, the gradient of each weight is clipped to be no higher than this value.

Parameters:

clipValue - the gradient clipping value.

setClipNorm

public void setClipNorm(double clipNorm)

Sets the gradient clipping norm. If clip norm is set, the gradient of each weight is individually clipped so that its norm is no higher than this value.

Parameters:

clipNorm - the gradient clipping norm.

getLearningRate

public double getLearningRate()

Returns the learning rate.

Returns:

the learning rate.

getMomentum

public double getMomentum()

Returns the momentum factor.

Returns:

the momentum factor.

getWeightDecay

public double getWeightDecay()

Returns the weight decay factor.

Returns:

the weight decay factor.

getClipValue

public double getClipValue()

Returns the gradient clipping value.

Returns:

the gradient clipping value.

getClipNorm

public double getClipNorm()

Returns the gradient clipping norm.

Returns:

the gradient clipping norm.

propagate

protected void propagate(double[] x,
 boolean training)

Propagates the signals through the neural network.

Parameters:

x - the input signal.

training - true if this is in training pass.

backpropagate

protected void backpropagate(boolean update)

Propagates the errors back through the network.

Parameters:

update - the flag if update the weights directly. It should be false for (mini-)batch.

update

protected void update(int m)

Updates the weights for mini-batch training.

Parameters:

m - the mini-batch size.

setParameters

public void setParameters(Properties params)

Sets MLP hyperparameters such as learning rate, weight decay, momentum, RMSProp, etc.

Parameters:

params - the MLP hyperparameters.