Ling 431/631: Corpus Linguistics

Ben Bergen

Meeting 11: Norming stimuli

October 29, 2007

Norm!

Linguistic stimuli to be presented to participants are used in most linguistic methodologies: laboratory-based, survey-based, elicitation-based, even introspection-based.

These stimuli, being by definition non-identical, will differ along a number of dimensions that may influence results, depending on the method. Some relevant dimensions of variation [vaguely from easiest to hardest to norm yourself]:

• independent frequency
• joint frequency [in a collocation, construction, etc.]
• [orthographic] length
• length [in syllables]
• relative frequency of homonyms
• relative frequency of different senses
• size of semantic, morphological, or phonological neighborhood
• concreteness
• imageability
• age of acquisition
• [spoken] length
• reading time in isolation

You can probably come up with tasks in which each of these might be factors.

Work that's been done for you

The MRC psycholinguistic database [http://www.psych.rl.ac.uk/] Contains 150837 words with up to 26 linguistic and psycholinguistic attributes for each:

Number of letters

Number of phonemes

Number of syllables

Brown written frequency

Brown number of categories

Brown number of samples

T-L written frequency

Brown verbal frequency

Familiarity rating

Concreteness rating

Imagability rating

Meaningfulness [Colorado]

Meaningfulness [Pavio]

Age of aquisition rating

Type [variant of other word]

POS [10 categories]

Common POS [N/V/Adj/Other]

Alphasyll [pref/suff/abbrv/hyph]

Status [colloquial/dialect/alien]

Variant phoneme

Capitalization of words

Irregular plural

Phonetic transcription

Edited phonetic transcription

Stress pattern

The English Lexicon Project [http://elexicon.wustl.edu/] includes the length, frequency, orthographic neighborhood size, bigram frequency, pronunciation, length [in phonemes, syllables, and morphemes], POS, as well as:

• Naming task [mean RT and accuracy]
• Lexical decision task [mean RT and accuracy]

Adam Kilgariff's BNC word frequency counts [http://www.kilgarriff.co.uk/bnc-readme.html] provide:

• A lemmatized frequency list for 6,318 words with more than 800 occurrences in the whole BNC. The definition of a 'word' approximates to a headword in an EFL dictionary such as Longman's Dictionary of Contemporary English: so, e.g., nominal and verbal "help" are listed separately, and the count for verbal "help" is the sum of counts for verbal 'help', 'helps', 'helping', 'helped'.

sort-order	frequency	word	word-class
5	2186369	a	det
2107	4249	abandon	v
5204	1110	abbey	n
966	10468	ability	n
321	30454	able	a

• Unlemmatized frequency lists for POS-wordform pairs
• Mean and variance for POS-wordform pairs across texts within the BNC

Norm it yourself

Some things you can do

• You already know how to get wordform frequencies.
• Type frequencies are harder, since you currently have to either define searches for individual lemmas by hand or do arithmetic on a frequency list by hand. [But could automate some solutions]
• Joint frequencies - collocational or collostrutional are also tougher because they need to be done by hand, unless, again, you automate the process. [Stefan Gries has a script to do this automatically, that runs in R http://www.linguistics.ucsb.edu/faculty/stgries/research/overview-research.html]
• Orthographic length can be computed using a very simple Excel formula: =LEN().
• Phonological length requires a phonological transcription
• You can automate syllable counting if you're very patient.

Statistical Analysis

The goal of norming is to determine that two or more sets of stimuli are not significantly different along some dimension or set of dimensions.

• To test whether there is a significant difference, you will usually use a t-test. A t-test is a statistical test used to determine the likelihood that two sets of continuous values are drawn from the same population
• You want to be pretty confidant that there's no significant difference, so the p value should be higher than 0.1.
• Why would we use a higher threshold for p in this case than the 0.05 we've used elsewhere?