Ling 640G: Cognitive Linguistics

Ling 423/640G: Cognitive Linguistics

Ben Bergen

Meeting 9: Experiment Design

September 23, 2008

Today we'll walk through how to design experimental research in language and cognition.

WARNING: There is a lot more to experiment design than what you will find in this introduction. I recommend before you begin to conduct large-scale work for public dissemination that you consult more thorough publications, like the paper assigned for today or Winer et al. (1991) or Kirk (1982).

1. Define your research question.

Your research question is the reason for running the experiment in the first place. Some criteria.

(a) The research should be novel - the answer to the question should not be known at present

· In other words, your question should not be answerable by looking up previous research.

(b) The answer to the question should be interesting because it bears on some broader issue.

So it shouldn't be Why does my roommate sound dumb when he talks? because of the limited audience it would interest.
But it could be What speech characteristics contribute to perceived intelligence? because sociolinguistics, as well as people in speech pathology, education, etc. might be interested in this question.

So it can't be Are undetectable space aliens making me say things? because there's no way to measure what undetectable space aliens are doing.
It can be Are people making me say things using lasers pointed at my brain?, assuming that you can measure whether lasers are pointed at your brain.

(d) Your research question must be situated relative to the current state of knowledge. For instance: (1) A particular theory makes a prediction that another does not – which is right? (2) Previous empirical work on the topic fails to provided a convincing account, due to methodological or other difficulties. (3) You can reason out a prediction from first principles.

· So it shouldn't be Do children innately know what the word 'dog' means? because no theory predicts this, nor can it be arrived at from reasoning.

· But it could be Do children innately know what a Noun is? because theories make different claims

So here’s a sample question:

When people process language about other people experiencing emotions, do they experience the described emotions?

This would be a good research question:

(a) We don’t know the answer, because not enough research has been done on it.

(b) Answering it could tell us about how we extract meaning from language.

(d) Current work shows that people experience perceptual and motor details of described scenes

2. Operationalize your question

Once you know what your question is, you need to identify a way to test in through careful manipulation. This manipulation requires the definition of at least one independent variable or factor and at least one dependent variable or dependent measure.

The dependent variable is the thing being affected, which can be directly and reliably measured. For example, this could be:

Reaction time to press a button.
Reading time for a word or sentence
Number of words remembered
Number of times a particular word is used
… among many others

The independent variable is the thing that is affecting the dependent variable. For example:

Whether or not a laser is pointed at your brain
Age, gender, handedness, etc. of the speaker
Native language of the experimental subject
Whether an image and a verb match
Whether a person had a large or a small dog as a child

Once you know your variables, you can rephrase your research question as a pair of hypotheses.

The experimental hypothesis is the claim that you want to verify the truth of - namely, that the independent variable measurably influences the dependent variable. For example, that laser beams increase cursing, or that slow speech makes people appear less intelligent.
The null hypothesis is that there will be no relation between the variables.

You are testing for evidence for the experimental hypothesis and against the null hypothesis.

Next time, we'll look at how statistics let us make conclusions about these two hypotheses.

3. Design details

Participants and items

Any experiment is conducted with a number of participants [the people taking part], using a number of items [the things they're exposed to].

What should participants be like?

Shouldn't know the intent of the experiment [it's OK to lie, as long as it hurts no-one].
Must agree to participate, and be informed of any risks and anonymity procedures
Must not differ in important ways across conditions [so you shouldn't have all old people in one condition and young people in another condition if that's not what you're testing.]

What should items be like?

Similar to each other as much as possible - pictures should be the same size, brightness, etc.; words the same frequency and length, etc. This is true both within and across conditions.
You will sometimes also want to include filler items, which are not related to the experiment, and basically serve as a buffer, so the subject doesn't catch on the manipulation

How many do I need of each?

You will usually want 5-20 observations per participant, per condition
So the number of items you need total will depend on the number of conditions you have
And for a full-fledged experiment, depending on the size of the effect, you could need as few as 6 or as many as 100 subjects.

Within or between participants?

Your independent variable[s] can be manipulated either within or across participants.

In a within-participants design, each participant is exposed to each condition of the independent variable. E.g. If you're looking at the influence of alcohol consumption on speech volume, each participant can be first given a placebo and asked to pronounce words, and then be given alcohol and asked to pronounce words.
In a between-participants design, each participant is exposed to only one condition of the relevant independent variable. E.g. each participant is randomly assigned to the alcohol or no-alcohol condition, and is asked to pronounce words.

Advantages of within-participants design:

Controls for inevitable differences across participants
This gives statistical tests more power in within-participants designs

Advantages of between-participants design

No carryover effects, where exposure to one condition affects the other condition.
No need for replication if you have small numbers of items
Sometimes, this is the only option, because you cannot manipulate the independent variable between participants, e.g. age [sometimes], intelligence [sometimes], or gender [sometimes]

Some experiments include both within- and between-participants factors. This perfectly acceptable.

4. Implement it

Your actual study can be high- or low-tech, depending on how finely you're manipulating stimuli and measuring responses. If you need millisecond timing, you have to use a computer.

Some typically used tasks that don't require a computer [and examples of what they can test]

Sorting objects or pictures into groups [categorization]
Putting objects in an order [representations of sequence]
Which are the same or which are different? [similarity]
Name all the things you remember [memory]
List features, properties, parts, subtypes, etc. [features, etc. of concepts or categories]
Name the first feature, property, part, subtype, etc. [prototypical aspects]
Rate similarity, likelihood, etc. on a [Likert] scale from 1-7 [similarity, reasoning, etc.]