Connectionist Language Modeling

Linguistics 431/631, Fall 2006

Moore Hall 228 and 153b (labs)

Ben Bergen

 

Overview

 

To model cognitive functions like language, perception, and memory, many scientists are turning to computational models that are based on the human brain. These connectionist models are gaining in popularity in Linguistics, Psychology, and Computer Science, as well as in other disciplines.

 

This course is an introduction to the connectionist modeling of language. The approach is extremely hands-on. Students will learn to use existing connectionist simulation programs to design models of human linguistic knowledge and functioning. The course is intended for students with little to no computer knowledge. NO PROGRAMMING EXPERIENCE IS REQUIRED.

 

This course is open to all advanced undergraduate students and all graduate students, in any discipline.

 

Coursework and grading

 

Most weeks consist of one regular class meeting and one computer lab meeting.

 

The coursework is composed of computer lab exercises and a term project.

 

Logistics

 

Please take full advantage of my office hours, currently scheduled for Monday 10:30-11:30 and Wednesday 1:30-2:30. If you cannot make these times, please let me know and we can schedule another time to meet. I am also available by email at: bergen@hawaii.edu. Lecture notes, an up-to-date course schedule, and links to online versions of course readings will appear through the semester at: http://www2.hawaii.edu/~bergen/ling631/


Schedule

 

Week

Date

Topic

Reading

1

8.22

Introduction

R1

 

8.24

Neurons and neurodes

R2

2

8.29

Computer lab: A first connectionist simulation

R3

 

8.31

No class – ICCG

 

3

9.5

The hidden layer and intermediate structure in the brain

 

 

9.7

Computer lab: Multi-layer networks

R4

4

9.12

Learning in the brain and connectionist networks

R5

 

9.14

Computer lab: Connectionist learning

R6

5

9.19

Morphology

R7

 

9.21

Computer lab: Morphology

 

6

9.26

Problems with morphological models

R8

 

9.28

Computer lab: Morphology II

 

7

10.3

Solutions in morphology

R9

 

10.5

Computer lab: Morphology III

 

8

10.10

Speech production

R10

 

10.12

Computer lab: Speech Production

 

9

10.17

No class – EMCL

 

 

10.19

Computer lab: Catch-up

 

10

10.24

Speech perception

R11

 

10.26

Computer lab: Speech Perception

 

11

10.31

More speech perception, Term project proposal due

R12

 

11.2

Computer lab: Speech perception II

 

12

11.7

No class - Election Day

 

 

11.9

Syntax

R13

13

11.14

More syntax

 

 

11.16

Computer lab: Syntax

R14

14

11.21

Computer lab: Syntax II

 

 

11.23

No class - Thanksgiving

 

15

11.28

The brain

R15

 

11.30

What is connectionism?

R16

16

12.5

Student presentations

 

 

12.7

Student presentations and wrap-up

 

 

12.12

Term project due

 

 

Readings

 

[Some papers require a login and password.]

 

R1       Overview of connectionism from Wikipedia: http://en.wikipedia.org/wiki/Connectionism

 

R2       Pages 1-10 of Kim Plunkett and Jeffrey Elman. 1996. Exercises in rethinking innateness. Cambridge, MA: MIT Press. http://www2.hawaii.edu/~bergen/ling631/papers/eri.pdf

 

R3       Get JavaNNS for your system: http://www-ra.informatik.uni-tuebingen.de/downloads/JavaNNS/

            The manual is included. Read sections 1-6

 

R4       Review section 6 of the JavaNNS manual

 

R5       Pages 10-18 of Kim Plunkett and Jeffrey Elman. 1996. Exercises in rethinking innateness. Cambridge, MA: MIT Press. http://www2.hawaii.edu/~bergen/ling631/papers/eri.pdf

 

R6       Sections 7 and 8 of the JavaNNS manual

 

R7       Rumelhart, D. and McClelland, J. (1986). On learning the past tenses of English verbs. In J. McClelland, D. Rumelhart, and the PDP Research Group (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, volume 2. pp. 216-271.
http://www.cnbc.cmu.edu/~jlm/papers/PDP/Chapter18.pdf

 

R8       Pinker, S. and A. Prince (1988). On Language and Connectionism: Analysis of a Parallel Distributed Processing Model of Language Acquisition. Cognition, 28:73-- 193. http://www.ecs.soton.ac.uk/~harnad/Papers/Py104/pinker.conn.html

 

R9       MacWhinney, B., & Leinbach, J. (1991) Implementations are not conceptualizations: Revising the Verb Learning Model. Cognition, 29, 121-157. http://psyling.psy.cmu.edu/papers/neuralnets/cog91.pdf

 

R10     Dell, G., F. Chang, and Z. Griffin (1999). Connectionist Models of Language Production: Lexical Access and Grammatical Encoding. Cognitive Science 23 (4): 517-542. http://www2.hawaii.edu/~bergen/ling631/papers/delletal.pdf

 

R11     McClelland, J. L. and Elman, J. L. (1986). The TRACE Model of Speech Perception. Cognitive Psychology, 18, 1-86.
http://www.cnbc.cmu.edu/~jlm/papers/McClellandElman86.pdf

 

R12     Gaskell, M. and W. Marslen-Wilson (1999). Ambiguity, competition, and blending in spoken word recognition. Cognitive Science 23:4: 439-462 http://www2.hawaii.edu/~bergen/ling631/papers/gaskellmarslen.pdf

 

R13     Elman, J. L. (1991). Distributed representation, simple recurrent networks, and grammatical structure. Machine Learning, 7, 195–225. ftp://ftp.crl.ucsd.edu/pub/neuralnets/machine_learning.pdf

 

R14     Frank, R., D. Mathis and W. Badecker. 2004. The Acquisition of Anaphora by Simple Recurrent Networks. Ms.
http://www.cog.jhu.edu/faculty/frank/papers/SRN-note.pdf

 

R15     http://vv.carleton.ca/~neil/neural/neuron-a.html

http://faculty.washington.edu/chudler/cells.html

http://users.rcn.com/jkimball.ma.ultranet/BiologyPages/N/Neurons.html

 

R16     Gurney, K. (m.s.) Notes on An Introduction to Neural Networks  http://www.shef.ac.uk/psychology/gurney/notes/l1/l1.html