Linguistics 431/631: Connectionist language modeling

Ben Bergen

 

Meeting 3: Multiple layers

September 5, 2006

 

Single-layer networks

á      A restricted class of input-output relations can be produced by networks with only one layer of input and one layer of output.

á      This class includes linear functions, sigmoid functions, and logistic functions.

á      What types of function cannot be produced?

 

Controversy

á      Minsky and Papert (1969) wrote ÒPerceptronsÓ, in which they argued that connectionist networks are not sufficient for capturing human cognitive functions.

á      Humans can reason using XOR

XOR

Input1

Input2

Output

0

0

0

0

1

1

1

0

1

1

1

0

á      Single-layer networks cannot

 

But introducing multiple layers of computation can solve this problem.

á      For XOR, introducing multiple layers allows us to solve the input-output function

á      XOR is still not possible with just one intermediate (hidden) node. Why?

á      But how about with two intermediate nodes? Consider this network:

á      What purposes could 1 and 2 serve that would allow 3 to yield the right outputs?

á      What weights would produce this behavior?

Multiple layers in the brain

 

How many layers of connections do we find between input to and output from an in vivo neural system?

á      The simplest types of neural circuit that include both input and output are those that control reflexes, like the knee-jerk reflex, or the blink reflex.

á      More complex ones like those responsible for language recognition and production may have on the order of 100 or 1000 intermediate layers

 

The knee-jerk reflex involves only a few layers of neurons.

            

á      When the thigh is caused to stretch which one can do by hitting the knee below the patella), a signal is sent through a sensory neuron in the thigh to the spinal cord.

á      This sensory neuron connects to interneurons in the spinal cord

á      The interneurons have excitatory connections to excitatory and inhibitory interneurons that excite and inhibit the extensors and flexors appropriately.

For a slightly more complex circuit, consider the auditory pathway.

á      Sound waves hit the tympanic membrane (ear drum), which moves the bones of the middle ear, which get the fluid in the cochlea vibrating.

á      This vibration of the cochlea, which is organized tonotopically, sets the cilia (hair cells) in motion, which makes them active.

á      The cilia then have synapses on the cochlear nuclei, and so on as in the image (which is from http://www-ece.rice.edu/~dhj/pathway.html)

á      In total, there are between five and seven layers of neurons intervening between the input and a signal reaching the temporal lobe of the cerebral cortex.

 

 

 

 

We can get a glimpse at how many intermediate layers are involved in a particular input-output relation from the time it takes information to flow through the system.

á      In the knee-jerk reflex, there are only four layers of neurons involved and the response takes about 50 msec (although thereÕs a little bit of time taken up by the activation of the stretch detector).

á      In audition it takes 50-100msec, for a signal to reach cortex

á      In general, one can expect each layer of neural structure to add about 1 to 5 msec to a response, plus time for the input and output interfaces to work.