Linguistics 431/631: Connectionist language modeling

Ben Bergen

 

Meeting 2: Neurons and neurodes

August 24, 2006

 

Neurons

 


Structure of the neuron

 

Each neuron can be seen as an information processing unit


 

 

 


 

Neurodes

 

Neurodes are like neurons

 

Neurons                       

Neurodes

Firing rate

Activation

Synapses

Connections

Synapse efficiency

Connection strength or weight

Excitatory/inhibitory synapses

Positive/Negative connections

 

Neurodes (mostly) perform two functions

 


Input summation is a very simple process. For a given node i:

 

For example, take a node i, which has inputs from two other nodes, g and h. The activations of g and h are 2 and 0.5, respectively. Their connection weights are 0.1 and 2, respectively. What is the sum of the inputs to node i? What would the activation be if the connection weights were 0.1 and 2?

 

The second step is the passing of this sum of products to the activation function. In principle, any function is possible, but in practice, only a small number are used.

 

Sigmoid functions in connectionist models have the following properties

 

Some more exercises:

      What would the activation of our node i be if it had the sigmoid function shown below?

      What would the activation be with inputs 1 and 1 and a sigmoid neurode?

      How could you change the weights such that the activation resulting from inputs 1 and 1 was 1?

            Activation

Net input

 


More about neurodes

 

Pattern matching

      Imagine that we want a network to perform a really simple function, like output=input.

      For example, imagine we want the following inputs to give rise to the corresponding outputs:

IDENTITY

Input

Output

0

0

1

1

      Easy with linear activation function, but how about sigmoid? If so, how? If not, why not?

 

Bias nodes

      Sometimes its useful to give nodes default activations other than 0 (if linear) or .5 (if sigmoid).

      E.g., in the case above with a sigmoid node, 1 needs to have default activation of 0. What does the sum input have to be for this activation to be produced by the activation function?

      We can solve this problem by implementing another node, a bias node, which always has an activation of 1 and which may be connected to any given node with some weight.

      What bias node weight would give the right sum input to node 1 when the input is otherwise 0?

      What does the strength of the input to node 1 from i1 have to be to overcome the bias?

 

Lets see how OR can be modeled, also using bias nodes

      There are two inputs and one output the basic idea is that if either input is 1, the output is 1. Otherwise the output is 0.

      The input-output function is as follows:

OR

Input1

Input2

Output

0

0

0

0

1

1

1

0

1

1

1

1

      Can this function be modeled with no bias, using either a linear or a sigmoid node? Why (not)?

      What should the weights of the three connections to node 1 be?