Linguistics 431/631: Connectionist language modeling

Ben Bergen


Meeting 3: Multiple layers

September 5, 2006


Single-layer networks

      A restricted class of input-output relations can be produced by networks with only one layer of input and one layer of output.

      This class includes linear functions, sigmoid functions, and logistic functions.

      What types of function cannot be produced?



      Minsky and Papert (1969) wrote Perceptrons, in which they argued that connectionist networks are not sufficient for capturing human cognitive functions.

      Humans can reason using XOR

















      Single-layer networks cannot


But introducing multiple layers of computation can solve this problem.

      For XOR, introducing multiple layers allows us to solve the input-output function

      XOR is still not possible with just one intermediate (hidden) node. Why?

      But how about with two intermediate nodes? Consider this network:

      What purposes could 1 and 2 serve that would allow 3 to yield the right outputs?

      What weights would produce this behavior?

Multiple layers in the brain


How many layers of connections do we find between input to and output from an in vivo neural system?

      The simplest types of neural circuit that include both input and output are those that control reflexes, like the knee-jerk reflex, or the blink reflex.

      More complex ones like those responsible for language recognition and production may have on the order of 100 or 1000 intermediate layers


The knee-jerk reflex involves only a few layers of neurons.


      When the thigh is caused to stretch which one can do by hitting the knee below the patella), a signal is sent through a sensory neuron in the thigh to the spinal cord.

      This sensory neuron connects to interneurons in the spinal cord

      The interneurons have excitatory connections to excitatory and inhibitory interneurons that excite and inhibit the extensors and flexors appropriately.

For a slightly more complex circuit, consider the auditory pathway.

      Sound waves hit the tympanic membrane (ear drum), which moves the bones of the middle ear, which get the fluid in the cochlea vibrating.

      This vibration of the cochlea, which is organized tonotopically, sets the cilia (hair cells) in motion, which makes them active.

      The cilia then have synapses on the cochlear nuclei, and so on as in the image (which is from

      In total, there are between five and seven layers of neurons intervening between the input and a signal reaching the temporal lobe of the cerebral cortex.





We can get a glimpse at how many intermediate layers are involved in a particular input-output relation from the time it takes information to flow through the system.

      In the knee-jerk reflex, there are only four layers of neurons involved and the response takes about 50 msec (although theres a little bit of time taken up by the activation of the stretch detector).

      In audition it takes 50-100msec, for a signal to reach cortex

      In general, one can expect each layer of neural structure to add about 1 to 5 msec to a response, plus time for the input and output interfaces to work.