NumModel2LayerNetBC - Two layer feed forward neural network implementation class

Developer Documentation

Centre for Vision, Speech & Signal Processing

Comments:
Simple neural network class. The network constructed is fully connected, feed-forward and 2-layer (i.e. inputs, hidden-layer, output-layer). At present only the back-propagation training algorithm has been implemented. More algorithms will follow.

Parent Classes:

NumModelBC

Variables:

WeightMatrixC flw;: First layer weights
NetVectorC flb;: First layer bias
WeightMatrixC slw;: Second layer weights
NetVectorC slb;: Second layer bias
IntT hu;: number of hidden units
NetVectorC _nvInputs;: Vector holding current input values into network
NetVectorC _nvHiddenLayerOutputs;: Vector holding current hidden unit output values
NetVectorC _nvOutputs;: Vector holding current output state of network
NetVectorC _nvTarget;: Vector holding current target vector (if any) for input
BooleanT _bInitialise;: Flag determining whether network should be randomnly initialised before calling the training algorithm Net Options
--------------------------------------------
IntT cycles;: Number of cycles
RealT lr;: Learning rate for network
RealT mom;: Momentum rate for network
RealT low;: Lower bounds for initilisation
RealT high;: Higher bounds for initilistation
BooleanT penaltyTerm;: Do we want to use the penalty term whilst training
BooleanT verbose;: Do we want to train verbosely
TrainMethod trainAlg;: Which training algorithm should we use

Methods:

NumModel2LayerNetBC()

Constructs empty instance

NumModel2LayerNetBC(const WeightMatrixC & flw,const NetVectorC & flb,const WeightMatrixC & slw,const NetVectorC & slb)

Constructs from matrices etc

NumModel2LayerNetBC(istream & in)

Construct from the input stream

NumModel2LayerNetBC(const NumModel2LayerNetBC & in)

Constructs a new network from an old network

BodyRefCounterVC & Copy() const

Make a copy of network

const NumModel2LayerNetBC & operator=(const NumModel2LayerNetBC & net)

Assignment of big object

Methods required from NumModelC

VectorC Evaluate(const VectorC & input) const

Evaluates Y = f(x)

void Fit(const NumVVDataSetC & data)

Train the network

BooleanT Save(ostream & out) const

save the model to stream

void Initialise(const NumVVDataSetC & traindata)

Initialise the network structure on a dataset

void Train(const NumVVDataSetC & data)

Routine that trains the neural network on the dataset This assumes the dataset has been normalised

void InitialiseRandom()

Initialise the neural network structure with random values

void PresentInput(const VectorC & input,const VectorC & target)

Present a constant input/output pair across the network

void PresentInput(const VectorC & input)

Present an input and corresponding target output to the network

RealT InputError(const VectorC & in,const VectorC & target)

Compute the square sum error of a sample

RealT MeanSquareError(const NumVVDataSetC & in)

Compute the total square sum error of a sampleset

NetVectorC DeltaOutputs()

Compute the deltas for the output layer

NetVectorC DeltaHidden(NetVectorC & deltaOutU)

Compute the deltas for the hidden units

IndexT MaxOutput()

Return the index of the maximum valued current output

RealT PercentageCorrect(const NumVVDataSetC & in)

Calculate the percentage correct on a labelled dataset

void SetHiddenUnits(IntT num)

Set the number of hidden units to use

void SetInitFlag(BooleanT flag)

Set the number of hidden units to use

IntT InputSize() const

Return the input dimensionality

IntT HiddenUnitSize() const

Return the number of hidden units

IntT OutputSize() const

Return the output dimensionality

NetVectorC InputV() const

Look at the current input to the network

WeightMatrixC FirstLayerW() const

Return the first layer weights matrix

NetVectorC FirstLayerB() const

Return the first layer bias

NetVectorC HiddenOutputs() const

Return the outputs of the hidden layer

WeightMatrixC SecondLayerW() const

Return the second layer weight matrix

NetVectorC SecondLayerB() const

Return the second layer bias values

NetVectorC OutputV() const

The output of the network

NetVectorC TargetV() const

The target pattern

void on_line_Train(const NumVVDataSetC & in)

Train the network using bp on-line

void on_line_Train_WM(const NumVVDataSetC & in)

Train the network using bp on-line with momentum

void off_line_Train(const NumVVDataSetC & in)

Train the network using bp off-line (i.e. batch mode)

void off_line_Train_WM(const NumVVDataSetC & in)

Train the network using bpm off-line

void RemoveInputNode(IntT index,BooleanT delhu)

Deletes an input, including weights from the network

void RemoveHiddenNode(IntT index)

Deletes a hidden node, including weights and bias from the network

IntSArray1dC Mao(const NumVVDataSetC & dset)

Do pruning suggested by Mao

IntSArray1dC UnitSelectionAlg(const NumVVDataSetC & dset)

Perform average weight magnitude selection on a dataset. It assumes that the datasets have not been normalised

trainset	dataset of training vector pairs
verifyset	dataset used for cross-validation
AcceptableError	user sets what error they are prepared to accept
StopAt	if error level never reached then stop at this number of inputs

IntSArray1dC AWMS(NumVVDataSetC & trainset,NumVVDataSetC & verifyset,IntT StopAt,RealT AcceptableError)

Perform average weight magnitude selection on a dataset. It assumes that the datasets have not been normalised

trainset	dataset of training vector pairs
verifyset	dataset used for cross-validation
AcceptableError	user sets what error they are prepared to accept
StopAt	if error level never reached then stop at this number of inputs

void SetCycles(IntT cyc)

Set the number of epochs

void SetLearningRate(RealT learnrate)

Set the learning rate

void SetMomentum(RealT momentum)

Set momentum

void SetBounds(RealT lowValue,RealT highValue)

Set the bounds for random initialisation

void SetPenaltyTerm(BooleanT flag)

Penalise high weights and force low weights to zero

void SetVerbose(BooleanT flag)

Print out training progress to screen

void SetTrainingAlg(TrainMethod trainAlgorithm)

Set the training algorithm

IntT Cycles() const

Get the current number of cycles

RealT LearningRate() const

Get the current value of the learning rate

RealT Momentum() const

Get the current value of the momentum

RealT LowBounds() const

Get the low bounds of initialisation

RealT HighBounds() const

Get the higher bounds of initialisation

BooleanT PenaltyTerm() const

Get the status of the penalty term flag

BooleanT Verbose() const

Get the status of the verbose flag

TrainMethod TrainingAlg() const

Get the current training algorithm

Internal Object Representation

#include "amma/Num/NumModelB.hh"

void Initialise(const NumVVDataSetC & train): Initialises model to an untrained state
Must be overloaded in derived class that train incrementally and provides a mechanism for restarting training from scratch.
void Fit(const NumVVDataSetC & train): Allows function parameters to be selected to fit function to data
const StringC GetInfo() const: Prints derived class information
BooleanT Save(ostream & out) const: Writes object to stream, can be loaded using constructor

#include "amma/Num/NumFuncB.hh"

BodyRefCounterVC & Copy() const: Makes a deep copy and is virtual
As the copy constructor but is virtual so can be called from a base class reference.
void SetSizeX(UIntT X): Sets size of input vector
void SetSizeY(UIntT Y): Sets size of output vector
VectorC Apply(const VectorC & X): Evaluate Y=f(X) as a process
VectorC Evaluate(const VectorC & X) const: Evaluate Y=f(X)
This is the main function that does all the work and must be overloaded in derived classes.
DListC<VectorC> Evaluate(const DListC<VectorC> & listX) const: Evaluate Y=f(X) for a list of X
This is achieved by iterating through the list of X and using the Evaluate member function that takes a single X. For sophisticated applications where speed is important, this function can be overloaded to do batch processing.
VectorC operator()(const VectorC & X) const: Evaluate Y=f(X)
MatrixC Jacobian(const VectorC & X) const: Calculate Jacobian matrix at X
Performs numerical estimation of the Jacobian using differences. This function has and should be overloaded for all cases where the Jacobian can be calculated analytically.
const StringC GetInfo() const: Derived class information
const StringC & GetName() const: Derived class type
UIntT SizeX() const: Size of input vectors
UIntT SizeY() const: Size of output vectors
BooleanT Save(ostream & out) const: Writes object to stream, can be loaded using constructor

#include "amma/DP/Process.hh"

VectorC Apply(const VectorC &): Apply operation.
IntT ApplyArray(const SArray1dC<VectorC> & in,SArray1dC<VectorC> & out): Apply operation to an array of elements.
returns the number of elements processed.
BooleanT Save(ostream & out) const: Save to ostream.
const type_info & InputType() const: Get input type.
const type_info & OutputType() const: Get input type.
BooleanT Save(ostream & out) const: Save to ostream.
const type_info & InputType() const: Get input type.
const type_info & OutputType() const: Get input type.
ProcTypeT OpType() const: Operation type lossy/lossless.
BooleanT IsStateless() const: Is operation stateless ?

#include "amma/DP/Entity.hh"

BooleanT Save(ostream & out) const: Save to ostream.
BodyRefCounterVC & Copy() const: Creat a copy of this object.

#include "amma/BRefCntV.hh"

BodyRefCounterVC & Copy() const: Creat a copy of this object.
BooleanT operator==(const BodyRefCounterVC & oth) const: Compair identitys.
BooleanT operator!=(const BodyRefCounterVC & oth) const: Compair identitys.
BooleanT Save(ostream & out) const: Save to ostream.

#include "amma/BRefCnt.hh"

void AddReference(): Another reference to the object has been created.
void RemoveReference(): One reference to the object was deleted.
Locking scheme -------------- The object is possible to lock (to make constant). The scheme assumes that the object is created, it can be locked, after that it can only be destroyed. The locked object cannot be unlocked. This locking scheme is very useful during debugging, using assert() function, when it is necassary to check that object is treated as constant and the constancy is not violated by any casting or passing through the copy constructor of shared objects.
void SetConst(void) const: This locks the object.
Often objects you wish to lock are already const.
void SetConst(void): This locks the object.
BooleanT IsConst(void) const: Returns TRUE if the object is locked, ie. it is assumed to be constant.
BooleanT IsNotConst(void) const: Returns TRUE if the object is unlocked.
ie. there is no special information if the object is constant or not. Counter state information -------------------------
BooleanT ToBeDeleted() const: Returns TRUE if there is only one reference to the object and the whole object or its reference counting part can be deleted.
BooleanT ToBeDeletedRemoveIgnoreNoRemove(): Decrement refrence by 1 return true if this leaves no refrences to the object.
BooleanT ToBeDeletedRemove(): Decrement refrence by 1 return true if this leaves no refrences to the object. This also checks the NoRemove flag.
BooleanT BodyMightBeDeleted() const: Returns TRUE if the reference counted part of the object can be deleted, ie. flag NOREMOVE is false .
BooleanT IsCountZero() const: Are there any refrences left ?
BooleanT BRCPtrCanDeleteObject() const: Used by BRRCPtrC to establish if an object has ZERO refrences and can be deleted.
IntT Count() const: Returns the current state of the counter, ie. how many references to this object exist.
const BodyRefCounterC & operator=(const BodyRefCounterC & b): It has not meaning to assign object 'b' to this object because it would destroy a history of the object which is counted. So this is a dummy function.
BooleanT IsValidObject() const: Test if object is valid.
When amma check is disabled this always returns true.
BooleanT UserBitTest(IntT x) const: Test user flag.
void UserBitSet(IntT x,BooleanT setit = TRUE): Test user flag.
void UserBitZero(IntT x): Set bit to zero.
void ReportBRCError(char * Msg): Report error, used in BRCPtrC.
UIntT Hash() const: Hash on address of object.
void SetUndeletable(): Make object undeletable.
Usefull to prevent recursive deleting in graph structures. Only hackers need this function.
void ReportInvalidObject(char * Msg = 0) const: Tell user about validation failure.

#include "amma/RefCBase.hh"

LabelT Label() const: Returns the label of this reference counter.
The member function is useful mainly to recognize objects during debugging. The value of the label is uniquely defined pointer.

Programmer:Kieron Messer, Documentation by CxxDoc: Tue Mar 20 10:49:27 2001

NumModel2LayerNetBC

Two layer feed forward neural network implementation class

include	"amma/Num/NumModel2LayerNetB.hh"
User Level:	Default
Library:	NumModelNeuralNet
Example:	NetTest.cc
Section:	default.Kieron Messer
In Scope:	std