Information about Node (neural Networks)

An artificial neuron, also called semi-linear unit, Nv neuron, binary neuron or McCulloch-Pitts neuron, is an abstraction of biological neurons and the basic unit in an artificial neural network. The Artificial Neuron receives one or more inputs (representing the one or more dendrites) and sums them to produce an output (synapse). Usually the sums of each node are weighted, and the sum is passed through a non-linear function known as an activation or transfer function. The canonical form of transfer functions is the sigmoid, but they may also take the form of other non-linear functions, piecewise linear functions, or step functions. Generally, transfer functions are monotonically increasing.

Basic structure

For a given artificial neuron, let there be m inputs with signals x₀ through x_m and weights w₀ through w_m.

The output of neuron k is:

Where (Phi) is the transfer function.



The output propagates to the next layer (through a weighted synapse) or finally exits the system as part or all of the output.

History

The original artificial neuron is the Threshold Logic Unit first proposed by Warren McCulloch and Walter Pitts in 1943. As a transfer function, it employs a threshold or Heaviside step function taking on the values 1 or 0 only. The inputs and outputs are both binary. Edges have no weights, but are either excitatory or inhibitory. Excitation is summed to calculate the activation value, with each input contributing a 1 value (connections may be replicated to effectively multiply this by any integer). Inhibition is absolute rather than relative, meaning that a single active inhibitory input will force the output of a node to 0, regardless of the values of the excitatory inputs.

Types of transfer functions

The transfer function of a neuron is chosen to have a number of properties which either enhance or simplify the network containing the neuron. Crucially, for instance, any multi-layer perceptron using a linear transfer function has an equivalent single-layer network; a non-linear function is therefore necessary to gain the advantages of a multi-layer network.

Below, u refers in all cases to the weighted sum of all the inputs to the neuron, i.e. for n inputs,

where w is a vector of synaptic weights and x is a vector of inputs.

Step function

The output y of this transfer function is binary, depending on whether the input meets a specified threshold, θ. The "signal" is sent, i.e. the output is set to one, if the activation meets the threshold.

See: Heaviside step function

Linear combination

The output unit y is a linearly weighted sum of its inputs plus a bias term, similar to θ above, which is independent of the inputs.

Networks based on this formulation are known as perceptrons. Typically the above transfer function in its pure form would only be useful in a regression setting. For a binary classification setting, the sign of the output denotes the class predicted; in this case it is more sensible (and more convenient in the context of the learning algorithm) to consider positive outputs to be 1 and negative outputs to be 0, thus reducing the transfer function to that of the step function above, where .

See: Perceptron

Sigmoid

A fairly simple non-linear function, the sigmoid also has an easily calculated derivative, which is used when calculating the weight updates in the network. It thus makes the network more easily manipulable mathematically, and was attractive to early computer scientists who needed to minimise the computational load of their simulations.

See: Sigmoid function

Pseudocode Algorithm

The following is a simple pseudocode implementation of a single TLU which takes boolean inputs (true or false), and returns a single boolean output when activated. An object oriented model is used. No method of training is defined, since several exist. If a purely functional model were used, the class TLU below would be replaced with a function TLU with input parameters threshold, weights, and inputs that returned a boolean value.

class TLU defined as: data member threshold : number data member weights : list of numbers of size X function member fire( inputs : list of booleans of size X ) : boolean defined as: variable T : number T ← 0 for each i in 1 to X : if inputs(i) is true : T ← T + weights(i) end if end for each if T > threshold : return true else: return false end if end function end class

Spreadsheet Example

generated with :de:Wikipedia:Helferlein/VBA-Macro for EXCEL tableconversion V1.7<\hiddentext>>

		Input			Initial		Output						Final
Threshold	Learning Rate	Sensor values		Desired output	Weights		Calculated		Sum	Network	Error	Correction	Weights
TH	LR	X1	X2	Z	w1	w2	C1	C2	S	N	E	R	W1	W2
							X1 x w1	X2 x w2	C1+C2	IF(S>TH,1,0)	Z-N	LR x E	R+w1	R+w2
0.5	0.2	0	0	0	0.1	0.3	0	0	0	0	0	0	0.1	0.3
0.5	0.2	0	1	1	0.1	0.3	0	0.3	0.3	0	1	0.2	0.3	0.5
0.5	0.2	1	0	1	0.3	0.5	0.3	0	0.3	0	1	0.2	0.5	0.7
0.5	0.2	1	1	1	0.5	0.7	0.5	0.7	1.2	1	0	0	0.5	0.7
0.5	0.2	0	0	0	0.5	0.7	0	0	0	0	0	0	0.5	0.7
0.5	0.2	0	1	1	0.5	0.7	0	0.7	0.7	1	0	0	0.5	0.7
0.5	0.2	1	0	1	0.5	0.7	0.5	0	0.5	0	1	0.2	0.7	0.9
0.5	0.2	1	1	1	0.7	0.9	0.7	0.9	1.6	1	0	0	0.7	0.9
0.5	0.2	0	0	0	0.7	0.9	0	0	0	0	0	0	0.7	0.9
0.5	0.2	0	1	1	0.7	0.9	0	0.9	0.9	1	0	0	0.7	0.9
0.5	0.2	1	0	1	0.7	0.9	0.7	0	0.7	1	0	0	0.7	0.9
0.5	0.2	1	1	1	0.7	0.9	0.7	0.9	1.6	1	0	0	0.7	0.9

Note: Initial weight equals final weight of previous iteration.

Criticism

The Artificial neuron is criticized by Izhikevich for not being biologically realistic. Although this argument is technically correct, it is largely academic, as artificial neurons are not intended to perfectly model biological neurons, but rather to perform complex non-linear computations.

See also

• Neural network
• Perceptron
• ADALINE
• Biological neuron models
• Connectionism

Bibliography

• McCulloch, W. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 7:115 - 133.