Linguistics 431/631:
Connectionist language modeling
Ben Bergen
August 24, 2006
Structure of the neuron
Each neuron can be seen as
an information processing unit
Neurons

Neurodes

Firing rate 
Activation 
Synapses 
Connections 
Synapse efficiency 
Connection strength or
weight 
Excitatory/inhibitory
synapses 
Positive/Negative
connections 
Neurodes (mostly) perform
two functions
Input summation is a very
simple process. For a given node i:
For example, take a node i, which has inputs from two other nodes, g and h.
The activations of g and h are 2 and Ð0.5, respectively. Their connection
weights are Ð0.1 and Ð2, respectively. What is the sum of the inputs to node i? What would the activation be if the connection
weights were 0.1 and 2?
The second step is the
passing of this sum of products to the activation function. In principle, any
function is possible, but in practice, only a small number are used.
Sigmoid functions in
connectionist models have the following properties
Activation