Affective Elaborations of Boolean Search Instructions for Novices: Effects on Comprehension, Self-Confidence, and Error Type

It is clear that in the end user revolution, more and more novices will need instruction in online search principles and techniques.

While one-on-one instruction is surely the most effective approach, it's not feasible with shrinking resources and the tremendous growth in the number of remote access users.

Since many users must teach themselves how to access online systems, written search instructions have become more important, including:

* online Help facilities,

* system manuals, and

* point-of-use handouts developed by librarians.

In this study I was interested in monitoring the effects of online searching instructions written in two different versions.

One version of the written instructions is minimalist in style and focuses strictly on the basic cognitive information needed to solve a search problem and used venn diagrams to illustrate boolean operations. This was called the un-elaborated version.

The other version elaborates the instructions according to a taxonomy of speech acts developed in an earlier study and did not use venn diagrams. This is called the affectively elaborated version.

In earlier studies with end-users searching systems, I found that the complexity and interactivity of the search situation made it difficult to discover what they had learned from the instructions.

I decided to design a study that focused on the pre-search activities involved in analyzing a search query and composing a search statement.

This type of experimental set-up allowed me to slow down the search process so that I could examine specific component skills, for example: understanding and correctly applying the AND, OR, & NOT operators.

In this case, I was interested in measures of both boolean search logic and self-confidence as a searcher.

An earlier study with end users showed that self-confidence played a significant role in several aspects of search performance, such as:

* being more efficient,

* using fewer irrelevant search terms,

* retrieving more pertinent materials, and

* finding the instructions more helpful.

As a follow-up in the current study, I also obtained self-confidence ratings.

(76 undergraduate) subjects were randomly assigned to read one of the two versions of the instructions.

Afterwards they all took a written test which allowed me to assess:

* their understanding of boolean logic

* their confidence in the solutions they offered

* and their style in composing search statements

(# concepts, # words, # restrictions)

Results showed that the affectively elaborated version of the instructions produced higher scores in both boolean logic and self-confidence.

In other words, subjects who read affectively elaborated instructions for composing boolean search queries had significantly higher boolean logic scores (about 40%) and higher confidence scores (up to 16%) (Presented in detail in the paper in Tables 2 & T3).

Writing Effective Instructions

One of the questions dealt with in this research is whether the effectiveness of point-of-use instructions can be significantly improved by adding appropriate affective elaborations to the text of search instructions.

The content of these affective elaborations is evidently crucial, and should be guided by an informed knowledge of the searcher's cognitive and affective world.

I found that written instructions are powerful enough to produce significant differences in both boolean thinking measures and self-efficacy measures.

What is it about the instructions that is capable of producing such cognitive and affective effects? I used a taxonomy of Affective speech acts that was developed in an earlier study of novice searchers.

The taxonomy specifies to the instructional designer what speech acts to include by providing an inventory in three levels: giving orientation, giving advice, and giving reassurances. (See Table 1 in the published ASIS Proceedings Annual Meeting, 1995)See JASIS Home Page

TRANSPARENCY: Instruction Sample 1

Look at a fragment of the instructions that were written by consulting the Taxonomy of Speech Acts.

Note that a search statment appears at the top and below it an explanation.

The assertion that...

This speech act is advising (orienting) the user by...

(1) Advising: Giving feedback (what will happen if...)

To counteract their anxiety or fear of failure and build positive attitudes

(2) Orienting: Telling what is reasonable to expect

To reduce their anger, frustration, and disappointment and maintain reality check

(3) Orienting: Identifying common errors

SEARCH STATEMENT 6

(children who have been abused and punished) NOT self-esteem

[Note: this is not a good search statement because it contains natural language phrases ("who have been") and (1) will almost certainly give you zero articles. (2) This is because it is unlikely that the long phrase would appear verbatim anywhere in the records of the database.]

vSearch statement 6 is incorrect. It is important to avoid using natural language phrases in your search statements. (3) Be sure to look at the search statement and edit out any natural language phrase present. To correct search statement 6, you can use two steps:

TRANSPARENCY: Instruction Sample 2

The assertion that...

This speech act is advising (orienting) the user by...

(4) Advising: Rank ordering options or strategies

To counteract their anxiety or fear of failure and build positive attitudes

SEARCH STATEMENT 7, Step A:

(children AND abused AND punished) NOT self-esteem

SEARCH STATEMENT 7, Step B:

(child OR children) AND (abuse OR abused) AND (punish OR punished)

NOT self-esteem

[Note: (4) Search statement 7A is better than 6 since it eliminates natural language phrases and translates them into concepts. However, search statement 7B is better still since it provides for possible word variations which are likely to occur]

ERROR ANALYSIS

The subjects produced written search statements in the test which provided a fascinating opportunity to identify the types of errors they made.

TRANSPARENCY: Stumbling blocks

To summarize the majority of their errors, I would put it this way: searchers have to overcome two stumbling blocks in acquiring search skills:

* The first stumbling block traps novices into thinking in terms of non-probabilistic logic.

* The second stumbling block ensnares them in semantic leakage.

Both of these pose problems for navigation because they are interrelated.

In order to compose valid boolean search statements, subjects must have an accurate understanding of both the semantic and logical features of the problem.

Non-probabilistic logic ignores the formal requirements of boolean logic. Instead, searchers simply include as many terms in the search statement as they think relate to the query problem. Thus reducing to 0 the probability of retrieving anything.

Semantic leakage occurs when searchers produce a search query that represents their ordinary understanding of the "aboutness" of a topic and ignore the syntax of combining concepts in a search statement.

There is a boundary between the logical and semantic aspects that searchers must learn to keep intact. They must actively oppose any 'leakage' in the rational 'membrane' that separates the semantic from the logical functions in a search statement.

Here is an example from the test that shows how a novice, after reading the instructions, did not separate the semantic aspects from the purely logical.

TRANSPARENCY: Sample Test Question

The task was to:

Circle the concept(s) that must appear in every article retrieved by this strategy:

(driving behavior OR drivers) AND risk-taking

The subject inaccurately circled "driving behavior," and wrote the following explanation:

Because the interest is in driving and correlation to behavior, style, motive, etc., I feel that the circled concept must be in every article.

The subject is reasoning by similarity instead of reasoning with probability logic. The problem can be solved purely from the perspective of formal boolean logic, that is, (A OR B) AND C, which requires that concept C appear in all of the retrieved records.

Instead, semantic leakage about the topic of "driving" influenced this novice to ignore the necessary logical implications of the boolean operator by reporting which of these three terms are considered more central to the intent of the query.

TRANSPARENCY: Sample Test Question 2

Other subjects' explanations reveal more of the semantic leakage phenomenon:

Circles "driving behavior" and explains, "because it's the general area under which more specific areas pertaining to driving fall."

Circles "driving behavior" and explains, "because it covers the main topics."

These subjects focus on the semantic issue of the broader and narrower relationship among the terms in the statement. They seem to think that the most general term is the most important, and therefore must be included.

They are interpreting the instruction "must" to mean providing an adequate description of the subject to be searched.

This an ordinary everyday language meaning of "must", rather than the boolean "must" which is formal and probabilistic.

Another subject circled both "driving behavior" and "risk-taking" with the following explanation:

Risk-taking is a characteristic of driving behavior. Because of this relationship, both concepts will be with articles retrieved.

This example also reveals that the subject is not viewing the statement in a formal way, but is interpreting the topic covered by the statement while ignoring the boolean AND operator.

Errors Found in Search Statements

TRANSPARENCY: Errors 1

Other types of logical and semantic errors were uncovered when I examined the search statements novices composed for this query:

"Write a search statement for a search that will give you articles that are only about the types of people who park illegally in parking spaces reserved for the handicapped."

The first two errors are logical

1. Making a Boolean Inversion

*drivers AND illegal parking OR handicapped

The OR should be an AND

As you look at these, you'll note that many statements contain more than one type of error.

6. Not using a Boolean Operator

*illegally handicapped parking

In a system that requires the insertion of operators, this search would fail

TRANSPARENCY: Errors 2

The rest of the errors are semantic.

2. Neglecting Word Form Variations

*park AND illegally AND handicapped

Without "parking" it's unlikely that this search would succeed.

3. Using Common Natural Language

*(illegal driving behavior OR types of illegal parking) AND (types of drivers who park in handicapped stalls)

Novices commonly use long phrases extracted directly from query statements and from everyday speech. Needless to say, that will cause failure.

TRANSPARENCY: Errors 3

4. Neglecting Concepts

*drivers AND handicapped spaces

"Parking" is an important concept ignored by this statement.

5. Adding (ANDing) Unnecessary Concepts

*illegal parking AND handicapped AND personality

"Personality" restricts the statement unnecessarily at the beginning.

Most of these error categories represent semantic error types. This is to be expected, given the cultural complexity of language and meaning.

It's evident from these examples that novices make multiple semantic and logical errors in a given search statement.

Error analysis produces data that can be used in interface design and Help instructions. This data points to the need to address these potential errors in all forms of instructions for novices.

Conclusion:

The novice searcher's online information world is complex, and discovering its cognitive and affective details is essential for building true usability into information retrieval technology.

Assuming, along with cognitive psychologists, that inner decision-making processes are behavioral acts, it is important to attempt to measure and identify the assumptions that lead to making a boolean inversion or other search error.

Data such as this indicate what aspects of boolean thinking and search logic need to be specifically explained in instructions.

If the cognitive and affective details of searching become known to librarians and system designers, it would be possible to make true user-based adjustments to

* interfaces and online screens,

* instructional sessions,

* point-of-use instructions,

* search handouts,

* documentation, and

* online Help facilities.

Improving the instructional register will become more of a critical problem as information technology becomes more complex, while the categories of users extends to wider segments of society.

It may be useful for system designers and instructors to consider that novice end-users will always be with us, and in ever increasing numbers.


Relevant Bibliography
Diane Nahl Home Page