.
Go to the bottom
Diane Nahl Home page
Back to Chapter 1
CHAPTER 4
PRINCIPLES FOR WRITING AFFECTIVELY ELABORATED
USER-CENTERED INSTRUCTIONS
4.1 LIBRARY TERMINOLOGY
Reading instructions can be conceptualized from the perspective of a discourse
model of communication. Naismith and Stein (1989) apply Winograd's (1977)
cognitive science model to the problem of understanding how library users cope
with the specialized language of librarians. As applied to library handouts,
the elements of this model may be described as follows:
(1) Librarians use stored schemas (or mental constructs) to select
words and expressions that apply to a current context, such as some particular
point-of-use instructions.
(2) Librarians have a cognitive model of users based on prior
experience (e.g., the intended users' level of knowledge).
(3) Users interpret the text of the instructions using stored schemas
applied to the current context (e.g., searching instructions versus printing
instructions).
(4) Users have a cognitive model of librarians based on prior
experience (e.g., that librarians use special words such as index and
citation).
(5) Users make cognitive attributions (or decisions) and build an
appropriate mental representation of the system with which they are
interacting.
Naismith and Stein (1989, p.548) report an investigation in which they found
that college students misunderstand half of the library terms that are used in
handouts and reference interviews. These misconceptions are called
"communication barriers" and may be encountered in online help screens,
computer manuals, and searching instructions. To eliminate the communication
barrier that handicaps the user, librarians need to switch from a vertical to a
horizontal relationship with users. "In a vertical relationship, an individual
nurses his or her self-concept, maintaining a superior image, with negative
results for others. In a horizontal relationship, communication is positive
and nonthreatening" (Naismith and Stein, 1989, p.545).
4.2 CHARACTERISTICS OF TWO TYPES OF INSTRUCTIONS
Writers of point-of-use instructions are now called upon to switch from a
vertical to a horizontal model of interaction with users. This distinction has
been explored in detail by Belenky, et al. (1986) in a book on two ways of
communicating called separate and connected knowing. The
characteristics of separate knowing (sometimes called 'the academic game') may
be summarized as follows (pp.102-103):
(1) doubting, or playing the doubting game;
(2) claiming to be tough-minded and sporting an adversarial style;
(3) being opposed to anything that savors of subjectivism, thus preferring
impersonal expressions;
(4) assuming a procedural model of reasoned critical discourse;
(5) speaking a public language as if others were present;
(6) appealing to reason detached from feeling;
(7) maintaining a commitment to objective analysis and excluding personal
concerns or feelings;
(8) valuing and using rhetorical skills ("ceremonial combat").
In contrast to the above, connected knowing has the following
characteristics:
(1) valuing personal experience as trustworthy knowledge;
(2) using empathy to have access to other people's knowledge as vicarious
experience ("Seeing the other not in one's own terms but in the other's
terms.");
(3) going to the intimate rather than the impersonal level;
(4) preferring the informal and the unstructured;
(5) trying to understand why people do or feel the way they do;
(6) being concerned with the personal, the particular, and sometimes, the
petty;
(7) trying to build trust and cement connections;
(8) remaining non-judgmental;
(9) involving fusion and acceptance ("communal mode");
(10) engaging in collaborative explorations;
(11) seeing personality as enriching one's understanding of the other;
(12) constructing metaphorical extensions to span the distance between self and
other.
It is apparent that these two modes of communication provide a contrast: the
one, analytic and impersonal, the other, subjective and empathic. One turns
outward, the other inward. The connected mode is relational, because it is
relationship oriented, the separate mode is separating because it is task
bound. Ideally, one would want to integrate the two modes since both have some
strengths that are worth preserving. Evidence that such integration is
achieved by those who strive for it is presented by Belenky et al. (1986).
They refer to "constructed knowledge," as having the following characteristics
:
(1) integrating the two modes: weaving together the strands of rational
thought and emotive thought, of objective knowing and subjective knowing. One
woman described her attempt as, "You let the inside out and the outside in;"
(2) recognizing the inevitability of conflict and stress, rather than insisting
on uniformity;
(3) working with a context-sensitive model where meaning varies with the
situation or the person;
(4) trying to imagine themselves inside the other person and becoming sensitive
to the interior life of others;
(5) becoming "passionate knowers;"
(6) valuing "attentive caring" in understanding the written word;
(7) establishing a communion with what they are trying to understand, using the
language of intimacy to describe the relationship between the knower and the
known;
(8) communicating with an author and having rapport through a nurturing
reaction.
These observations are helpful to writers of point-of-use instructions since
they pinpoint techniques for maintaining rapport with novices "through a
nurturing reaction" to their anticipated needs. Affectively elaborated
instructions recognize the inevitability of stress and doubt. Through them,
writers can establish a communion with what novices are trying to
understand.
4.3 FACILITATION EFFECTS OF AFFECTIVELY ELABORATED INSTRUCTIONS
Comprehending point-of-use instructions requires (1) focusing on the
referent being discussed, (2) deducing the implications, and (3)
identifying the presuppositions. This process of understanding may be
seen from an example in Exhibit 4, above (Section 2.1.4):
You'll notice on the screen a list of terms that are synonyms of your own
words. When you feel that one of the official terms fits the topic you're
looking for, HIGHLIGHT DESIRED TERM(S) USING [up and down] ARROW KEYS and PRESS
ENTER TO SELECT TERMS. By pressing Enter you tell the computer
you're going to use that term for a search, but the search won't start until
you have finished highlighting all of the official terms that might fit your
topic. You can highlight and press Enter as many times as you wish to
have a term included in your search.
To comprehend the instruction "HIGHLIGHT DESIRED TERM(S) USING [up and down]
ARROW KEYS. PRESS ENTER TO SELECT TERMS" one needs to (1) focus on the
screen where a line is brighter than the others; (2) deduce that pressing
Enter will not start the search; and (3) be aware of the presupposition
that terms are to be highlighted and entered only when they fit the topic of
interest. These three discourse events are potential sources of confusion,
misinterpretation, and error. The expanded affective instructions attempt to
spell out these referents, implications, and presuppositions in order to reduce
the danger of information overload. One needs to investigate whether these
discourse expansions produce better comprehension and better performances. If
they do, what are the mechanisms by which this facilitation effect is achieved?
The following theoretical justifications may be given.
4.3.1 Mechanisms of Facilitation
1. Wordiness
More wordy instructions create more space and time between the critical
information bits supplied by the terse cognitive language. The end-user has
more psychological 'room' to analyze and assimilate the new information.
2. Stress Control
The panic or other emotional stress that comes with information overload, is
counteracted or kept under control by positive counteracting emotions that are
aroused by the affective speech acts (Bandura, 1989).
3. Goal Orientors
The learning task is more difficult when there are few goal orientors in
the instructions. The affective elements of the elaborated instructions
function as goal orientors, in the sense that they provide the learner with a
motivational and goal-oriented context for each cognitive item of information
in the instructions. In this sense, these affective elements provide selective
attentional focus, a feature that has been found to improve learning from
instructions or models (Bandura, 1989, p.51). These "attention directing aids"
can be structured to help observers or readers focus on pertinent features.
The language of "advice giving" captures this function better than the language
of technical or mechanistic description.
4. Hierarchy of Importance
The affective speech acts help create a hierarchy of importance of the
various cognitive sequences being defined or described in the instructions.
This strengthens the learning and deepens it for better recall and creative
generalization. As Bandura reports, much research exits to show that learners'
cognitive competencies and perceptual sets dispose them to look for some things
but not for others. Their expectations channel not only what they look for but
partly affect what features they extract from observations and how they
interpret what they see and hear (Bandura, 1986, p.53).
The affective speech acts exert control over expectations and perceptual sets.
They help weaken the power of idiosyncratic interpretations by channeling their
expectations in standardized directions.
5. Community Building
The affective component helps build affinity and solidarity with the community
of searchers. End-users can see themselves as a generational cohort in which
there are plenty of others sharing the same problems and profiting from the
same advice.
6. Rewardingness of Model
The affective component of instructions is warmer than the cognitive component,
and therefore, reading the instructions is more rewarding. The librarian
surrogate behind the traditional mechanistic instructions may not be as
rewarding to the user as the librarian model who is behind the affective
discourse. This affects a number of things, as much research has shown:
[Role] models who are integrated and otherwise rewarding tend to be sought out,
whereas those who lack attractive qualities are ignored or actively rejected.
... Different forms of modeling [including verbal descriptions] are not
equally effective. They may differ in the amount of information they convey
and in their power to command attention. How information is communicated can
also affect how it is cognitively processed (Bandura, 1986, pp.54, 71).
It is clear from this that the verbal modeling of affective behavior which
characterizes affectively elaborated user-centered instructions, has the
potential to make the reading of instructions more rewarding to the user. How
the user understands cognitive information is thus, in part, a function
of how rewarding or pleasant it is to read the instructions.
4.3.2 Affective Discourse/Cognitive Discourse
Affective discourse is designed to be more arousing than cognitive discourse.
Research has shown that "words that arouse emotion are widely used as vehicles
for affective learning. ... Remarks arousing positive reactions can be used to
foster likes and attractions" (Bandura, 1986, p.185). The process by which the
librarian or BI instructor can influence the happy disposition of patrons is
like the process of "vicarious affective learning" in teaching through
modeling:
In vicarious affective learning, events become evocative through association
with emotions aroused in observers by the affective expressions of others.
Observers get aroused by displays of emotion conveyed by vocal, facial, and
postural cues of models. Such affective social cues acquire arousal value
largely through correlated interpersonal experiences. That is, when others
close to one are happy, anxious, angry or despondent they are likely to behave
in ways that make those around them feel good or miserable. As a result of
contingent affective experiences, the emotional expressions of models generate
anticipatory arousal in observers (Bandura, 1986, p.186).
One can surmise that writing point-of-use instructions, or on disc
instructions, can be made more effective by the integrated addition of
affective speech acts that model desirable search traits such as a
happy disposition, optimistic expectations, appreciation
of system features, interest in decoding technical details, and so on.
A deeper understanding of the style required for user-centered language and
communication may be gained by applying humanistic principles in the writing of
instructions. An especially relevant approach is that of Carl Rogers, a noted
humanistic psychologist (Rogers, 1951; 1966; 1969).
4.3.3 Rogerian Principles
As discussed by Hergenhahn (1990, pp.439-43), a basic tenet of this approach is
that humans have an inborn need for positive regard, which is to receive
sympathy, care, and acceptance from relevant sources. Out of this need to
receive positive regard from others, we evolve the need for positive
self-regard, which manifests itself as holding favorable self-perceptions.
A problem arises when the positive regard others give us is constrained by
conditions of worth. For example, children learn that they can have
their parents' love and respect only if they behave in accordance with the
parents' wishes. This conditional acceptance helps develop the children's
conscience. At the same time, however, it leads to internalization of
unfavorable self-assessment. One's perception of self-worth suffers and
frequently leads to maladjustment such as alienation, anxiety, denial,
self-blame. Rogerian therapy attempts to remedy this state of mind by giving
clients unconditional positive regard so that they might experience
positive self-regard even in failure.
The basis of this remedy process is called nondirective therapy because
it encourages clients to solve their own problems within a proper atmosphere.
Because of this feature, Rogers called it client-centered therapy or the
person-centered approach. It involves the attempt to understand
clients' internal frame of reference, which includes their thoughts and
feelings in their struggle to cope. If we substitute "searchers" for clients,
and "librarians" for therapists, we can summarize the Rogerian form of
user-centered instructions in the following six principles:
(1) Searchers and librarians (or programmers and documentation experts) must be
in affective contact. That is, they must express to each other
the relevant feelings that the search environment evokes in both. Instructions
must therefore have an affective component explicitly dealing with the feelings
engendered by the search environment.
(2) Searchers must be perceived as being in a state of incongruence,
which is the need to succeed while lacking sufficient knowledge. This is
to be understood as a stressful mental state. Instructions must therefore
explicitly legitimize the searcher's negative psychological state by making it
a normal problem created by the setting requirements rather than the searcher's
fault.
(3) Librarians must be in a state of congruency in relation to
searchers. This intentionality takes the form of being accepting of searchers'
weaknesses and ineptitudes, and at the same time, having the desire to help
searchers acquire positive self-regard. The tone of this acceptance must be
clearly present in the instructions.
(4) Librarians must give searchers unconditional positive regard. How
instructions can best express this attitude or philosophy will no doubt be an
important future direction for this type of research.
Back to the top"
(5) Librarians must seek an empathic understanding of searchers'
internal frame of reference. User-centered instructions must take the
point-of-view of the searcher, not of the system. To do this adequately,
research will have to discover empirically what users' cognitive and affective
needs are while searching. A step in this direction may be found in recent
research on novice end-users by Tenopir et al. (1991; 1990).
(6) Searchers must perceive the fact that the librarians are giving them
unconditional positive regard and are attempting to understand empathically
their internal frame of reference. This is one important reason why
point-of-use instructions should regularly be rated by users as to their
helpfulness and comprehensibility.
4.4 CONTENT ANALYSIS OF INSTRUCTIONS: LOEX SAMPLE
A sample of CD-ROM instructions used in academic libraries was requested from a
national bibliographic instruction clearinghouse (Library Orientation Exchange
(LOEX), 1992). Of the two-dozen individual instruction sets received, seven
from different libraries were written for novices learning to use the various
databases available from the H. W. Wilson Co. The first step was to read each
sentence and categorize it into one of the three behavioral domains. The
purpose was to come up with an inventory (though not a complete one) of
instructional speech acts commonly occurring in CD-ROM library instructions.
The idea was not to develop a complete catalog of CD-ROM speech acts but to
inspect a range of them for the purpose of constructing a taxonomic continuum,
without specifying all of the steps on it. This is explained in detail.
The following is a variety of speech acts obtained from the seven LOEX
samples:
-- tells what happens if you don't select choice X
-- tells what happens if you select choice X
-- announces the overall topic of a section (as in a heading or
sub-heading)
-- unitizes overall task into numbered sub-tasks
-- identifies the condition when the command will do
something
-- identifies specific location in text (e.g., "(see the section on
How to Print on page 3")
-- defines terminology
-- identifies a key and its location on the keyboard
-- gives a specific example for some complex action
-- specifies which characters should not be used (e.g.,
that quotation marks are not to be typed)
-- alerts users to check command statements for typos
-- specifies subjects covered
-- reproduces what the screen should look like using a
diagram
-- advises users on how to make sure of something (e.g., "Look
at which line is highlighted...")
-- contrasts the essential difference between two search modes
-- introduces a related process (e.g., lateral searching or
explode)
-- explains the meaning of Boolean logic (e.g., with a Venn
diagram)
-- relates individual library holdings to indexed material in
database (e.g., warns that library does not subscribe to every
journal retrieved in a search)
-- specifies time coverage of indexed material in database
-- gives a tip
-- identifies fields on screen display
4.4.1 Inventory of Affective Speech Acts
A continuum was constructed for the affective domain since the strategy
that was experimentally investigated in this dissertation concerns the effect
of adding affective speech acts through elaborations. Future research should
investigate the other two domains. The affective continuum should have a lower
end (level 1) addressed to the negative emotions and anxieties which novice
users experience. Level 1 is equivalent to the "Orientation" phase of library
learning (Jakobovits and Nahl-Jakobovits, 1987, p.207). Once negative emotions
are taken care of, novices need to be given advice on how to proceed, both
generally and specifically. This is an "interaction" phase (level 2) and
attempts to build positive attitudes in novices by offering advice and sharing
convenient tips. Finally, the most "internalized" form of affective speech
acts (level 3) is expressed through giving reassurances to encourage acceptance
of the system and to overcome resistance to using it.
Table 2 presents the results of inventorying and categorizing the affective
speech acts found in the seven sample instructions. The function of
affective speech acts in CD-ROM instructions for novices includes three
levels: Orienting, Advising, and Reassuring. These three levels operate
simultaneously within the instructions, though instructions seem to vary
greatly in terms of the distribution of speech acts at each level, as well as
from each domain. As shown below, the LOEX sample tends to devote about equal
space to cognitive and affective speech acts. This taxonomy serves to compare
the LOEX instructions--called the unelaborated version--with the new
instructions in which affective speech acts have been inserted into the text of
the unelaborated version. Hence, the new text is called the affectively
elaborated version. The rationale for the two versions is presented
below.
4.4.2 Taxonomy of Affective Speech Acts
As has been discussed in the above review, the theoretical approach adopted
here consists in integrating three ideas from the literature:
(1) speech act theory (instructional speech acts);
(2) the taxonomy of the three behavioral domains (affective,
cognitive, sensorimotor);
(3) the method of text elaborations (adding sentences).
The basic idea from speech act theory is that each sentence and heading that
appears in the text of instructions can be considered as an instructional
speech act, that is, a verbalized intention of the writer whose purpose is to
explain something or to teach its use. A recent important source on speech act
theory is Habermas (1984). He reviews the work of Austin (1965) who originated
this line of verbal analysis, as well as the work of Searle (1969) who extended
and popularized Austin's typology.
Locutionary speech acts are ordinarily used to say something or to
express some state of affairs. Illocutionary speech acts refer to
verbal acts that accomplish a known interpersonal transaction, e.g., "I promise
I won't forget." Here, the illocutionary speech act is a promise. Other
illocutionary speech acts are performed in greeting someone, submitting an
application, or making a request. Everyday social life proceeds through the
endless cycles of illocutionary speech acts that constitute the fabric of
community life.
A third type of speech act is called perlocutionary and refers to
effects produced upon the hearer by the speaker. As Habermas puts it, a
perlocutionary speech act is "to bring about something through acting in
saying" (1984, p.288). For instance, librarians writing instructions for
novice users of CD-ROM databases, can say something that users take as
reassurance about their success and legitimacy. If the reader feels
reassurance, there is a perlocutionary effect. There is no guarantee that the
desired perlocutionary effect will be obtained, however, it is necessary to
discover speech acts that encourage novice searchers to make use of
instructions that are available to them. To obtain feedback on the
perlocutionary effect, searchers may be asked to what extent they find
instructions comprehensible, clear, motivating, or helpful. Those who state
that "these instructions are very helpful," show the perlocutionary results of
the librarian's intention to promote acceptance of the instructions and its
contents.
Point-of-use instructions and other help assistance systems can thus be defined
as a communicative exchange in which instructional speech acts of various types
are performed by librarians to regulate the behavior of library users and to
produce favorable psychological reactions in them. Habermas specifies the
distinction between illocutionary and perlocutionary acts as follows:
By means of an illocutionary act a speaker lets a hearer know that he wants
what he says to be understood as a greeting, command, warning, explanation, and
so forth. His communicative intent does not go beyond wanting the hearer to
understand the manifest content of the speech act. By contrast the
perlocutionary aim of a speaker, like the ends pursued with goal-directed
actions generally, does not follow from the manifest content of the speech act;
this aim can be identified only through the agent's intention" (Habermas, 1984,
p.290).
Thus, while illocutionary speech acts are "self-identifying" (Habermas, 1984,
p.290), having an internal, conventional format, perlocutionary effects are
external and situation bound. In terms of the taxonomic perspective adopted in
this dissertation, it appears that cognitive speech acts correspond to
illocutionary results while affective speech acts correspond to
perlocutionary effects. This may be seen in the distinction Habermas draws
between illocutionary and perlocutionary acts:
Perlocutionary acts are an indication of the integration of speech acts into
contexts of strategic interaction. They belong to the intended consequences or
results of a teleological action which an actor undertakes with the intention
of influencing a hearer in a certain way by means of illocutionary successes.
Naturally, speech acts can serve this nonillocutionary aim of influencing
hearers only if they are suited to achieve illocutionary aims" (Habermas,
1984, p.292-293).
Translating this into our taxonomic language of behavioral domains by
identifying "illocutionary" with "cognitive" and "perlocutionary" with
"affective," the idea occurs that affective speech acts require the integration
of cognitive speech acts into "contexts of strategic interaction." In other
words, librarians who are writing point-of-use instructions for novice users of
CD-ROM databases can be viewed as engaged in "strategic interaction" with
novice users. The cognitive speech acts are illocutionary explanations while
the affective speech acts are perlocutionary consequences reflecting the
librarians' intentions. Thus, the librarian is the actor who undertakes
teleological actions with the intention of influencing users by means of
illocutionary successes in the form of user-centered instructions. As Habermas
would say, librarians 'intervene in the world' of users to bring about
teleological actions (1984, p.293).
Cognitive speech acts, like 'illocutionary results,' "appear in the lifeworld
to which participants belong and which forms the background for their processes
of reaching understanding" (Habermas, 1984, p.293). The cognitive knowledge of
an operation system must be logical and mechanical, in conformance with the
hardware and the functions of its elements. In contrast, affective speech acts
are not part of the logical hardware system; they are part of users--their
preferences, feelings, and attitudes. Affective speech acts are personal,
drawing librarians and users into a psychological or emotional relation.
According to Habermas:
I would like to suggest that we perceive perlocutions as a special class of
strategic interactions in which illocutions are employed as means in
teleological contexts of action" (1984, p.293).
Thus, affective speech acts may be viewed as strategic interactions in
which cognitive speech acts are employed as means in writing point-of-use
instructions and other help assistance. Habermas seems to feel that the use of
deliberate perlocutionary effects in speech acts characterize strategic rather
than communicative actions (1984, pp.294-295). This raises a potentially
important problem for the future: Does the deliberate use of affective speech
acts in help assistance constitute some form of manipulation and deceit?
Habermas seems to say that all strategic behaviors depend on deceit, in
the sense that hearers would not be successfully affected by a ploy unless they
were unaware of the attempt. It seems that "deceit" may be too specific since
it unnecessarily implies negative intentions. The teacher-learner exchange
always implies that the teacher knows more than the learner. Thus, teachers
may intend to produce effects in learners in the absence of the learner's
knowledge of what the teacher is intending or planning. To that extent there
is the potential for deceit, but more usually, benefits are intended. At any
rate, future research needs to address the ethical implications of using
perlocutionary strategies in affective speech acts embedded in help assistance
instructions.
The instructional speech acts that make up the text of
instructions can be counted and categorized according to some scheme. The
taxonomic scheme adopted here consists of a tripartite division of user
behavior into three domains and three levels. The three
domains are the affective (having feelings, interests, purposes), the
cognitive (acquiring mental models, definitions, strategies), and the
sensorimotor (relating to displays, records, keys, formats). The text
of instructions would thus be expected to contain some affective speech acts,
some cognitive speech acts, and some sensorimotor speech acts.
The three levels represent phases of "internalization," a psychological term
referring to the degree to which individuals appropriate, or assimilate, a new
habit into the structure of the self (Kelman, 1958). Thus, level 1 is an
orientation phase during which novices have to assimilate basic
information which requires a re-orientation of their current approach to a
problem situation. Level 2 is an interaction phase during which novices
have to assimilate the meaning of new terms and the function of new procedures.
Level 3 is the internalization phase proper which allows novices to feel
empowered and perceive themselves no longer as novices, but instead as
independent searchers (even though it may be at a low level of skill). In
complex learning environments such as online catalogs and CD-ROM databases,
novices are expected to operate at all three levels and domains at the same
time, though in different sub-areas of the skills to be acquired.
The third element of the theory applies the idea of text elaborations to
categorized instructional speech acts. Thus, regular unelaborated instructions
can be analyzed for their distribution of speech acts and additional sentences
can be inserted to make up for under-utilization of some important category.
These elaborations may be in any of the three domains or levels. In this
research a limited objective was chosen, namely, to focus on the effects of
adding affective elaborations at the three levels. Future research will have
to investigate the other two domains. The three levels of affective
elaborations were defined as orienting, advising, and
reassuring (see Taxonomy in Table 2). The affectively elaborated text
used in this study contains more than twice as many affective speech acts as
cognitive speech acts. This greater than two-to-one ratio of the elaborated
text may be compared to a less than one-to-one ratio for the unelaborated text.
The experiment measured under controlled conditions, the effects produced in
the search behavior of novices by this difference in the affective/cognitive
ratio of instructional speech acts.
The affective taxonomy proposed in Table 2 can be considered a hypothesis to be
tested more fully in the future. Its theoretical assumptions are that users
perform affective behaviors during searching (along with the cognitive and the
sensorimotor), and that these affective behaviors ("feelings") can be
influenced by the speech acts in the instructions. The taxonomy represents
specific hypotheses regarding which types of affective behaviors users
experience (column 4) and what instructional speech acts can influence these
behaviors appropriately (column 5). For instance, feelings of impatience
(Category 1a in Table 2) can be reduced by telling how long some process will
take, thus eliminating blind waiting time. Or, feeling dissatisfied (Category
3e in Table 2) can be averted by pointing out the value of some result that the
user disregarded. The taxonomy can be used by librarians to identify potential
user feelings and provide for them in terms of appropriate speech acts in the
instructions. For the cognitive domain, it was found that only four
sub-categories were needed for this sample of instructions, all of them at
level 2 (see Table 3).
A variety of measures was used to insure the observation of important features
of search behavior. The results of this experiment show whether the affective
method of elaboration in CD-ROM instructions produces desirable effects in the
behavior of novice users, such as greater success and satisfaction, a more
adaptive search style, less frustration, stronger self-efficacy beliefs as a
searcher, and greater understanding. If the results are significant, a new
approach will have been found for designing user-centered instructions, namely,
the method of affectively elaborated texts. This method was applied to a
sample of the CD-ROM instructions obtained for this study from the LOEX
national clearinghouse. It is hoped that the findings of this study will allow
a more objective approach in the future, one that is more specifically guided
by a taxonomy and appropriate procedures for constructing instructional speech
acts.
4.4.3 Text of the Instructions
For the unelaborated version, one of the seven instructions received for the
Wilson databases was chosen at random to minimize selection bias. It is from a
LOEX participating library, dated August 1990 and is five pages long. Pages 1
and 2 have the text that explains the two search modes in the Wilson databases.
Page 3 contains four diagrams of screens with explanations. One of these was
included. Pages 4 and 5 contain instructions for downloading a CD-ROM search.
Only Pages 1, 2, and 3 were used in this research. They are reproduced in
Appendix I. To facilitate discussion, the sentences in the instructions have
been numbered, though they were not numbered for the subjects.
The text of the affectively elaborated version also appears in Appendix I. It
was prepared by examining every sentence of the unelaborated version and
inserting one or more affective elaborations as deemed advisable. While this
process is intuitively based, it is not different from what librarians have to
do when making up point-of-instructions.
By way of illustration, examine sentences 4, 5, and 6 for the
unelaborated instructions, (see Appendix I):
(4) BROWSE SEARCH
(5) This is the simplest method of searching. (6) BROWSE SEARCH allows you to
search a single subject.
As Table 4 shows, sentences 4, 5, and 6 were categorized using the taxonomy of
cognitive instructional speech acts (see Table 3) as:
Sentence 4 = 1h, Affective--Table 2: "Orienting by telling where that
topic is in the text" and 2a, Cognitive--Table 3: "Giving
descriptions, incl. titles, headings, labels)."
Sentence 5 = 2a, Affective--Table 2: "Rank ordering options or
strategies") and no Cognitive representation.
Sentence 6 = no Affective representation and 2d, Cognitive--Table
3:
"Defining the function or use of some system feature
(commands, keys, procedures)."
One might start by answering the question, Does the speech act have an
affective component? If Yes, at which level and sub-category? For
sentence 4, the answer is Yes, as can be seen from the affective taxonomy of
instructional speech acts (Table 2), where entry 1h seems to fit the
writer's purpose for this speech act, namely, "Telling where something needed
can be found such as sub-headings." In other words, the sub-title in sentence
4 orients the user to look in that section if they wish to find
information on how to do a Browse mode search, rather than somewhere else.
The second question is whether sentence 4 has a cognitive component.
Looking at the cognitive categories (see Table 3) we can see that this sentence
fits the cognitive category 2a, namely, "Giving descriptions, including
titles, sub-headings, labels, in the context of instructions." Sentence 5 does
not appear to have a visible cognitive component; its affective function fits
category 2a, namely, "Rank ordering options or strategies." Sentence 6
does not have an affective component; its cognitive category fits 2c,
that is, "Defining the function or use of some system feature (commands, keys,
procedures)."
4.4.4 Quantitative Comparisons Between Unelaborated and Elaborated
Instructions
Table 6 gives some quantitative comparisons between the unelaborated and
affectively elaborated versions. The elaborated version is about three times
longer in the total number of speech acts (91 vs. 31), but this difference is
due almost exclusively to the greater number of affective speech acts (82 vs.
22). This result is a direct outcome of the method of adding affective
elaborations to the existing text. The unelaborated text contained the same
number of affective speech acts as the cognitive (22 vs. 25). This was noted
as typical when the other instructions in the LOEX sample were examined. A way
of combining these two indices is to compute an A/C ratio, which would be, in
this case, .9 (22/25) in the case of the unelaborated version used here.
Ratios of around 1.0 would be expected for the unelaborated, system-centered
instructions as currently designed in U.S. libraries today. Future research
should show what optimal ratios are for different types of instructions and
audiences. In contrast, the elaborated version here constructed has 82
affective speech acts and 35 cognitive, which gives an A/C ratio of 2.3.
Clearly, the intended manipulation of the method worked well in this case by
more than doubling the A/C ratio.
CHAPTER 5
METHODOLOGY, DESIGN, AND PROCEDURES
5.1 CD-ROM DATABASE SYSTEM USED
The subjects in the study performed searches in the Readers' Guide
Abstracts (RGA), a CD-ROM periodical index database produced by the H.W.
Wilson Company. It indexes over 200 general interest magazines, and is
intended for students and the general public. The database is based on the
familiar print index Readers' Guide to Periodical Literature. RGA is
one of the databases provided by Wilson as part of their Wilsondisc CD-ROM
system. Wilsondisc currently comes with three different user interfaces,
including two for end-users and one for experts. Subjects in this study used
an earlier Wilsondisc version that had four user interfaces, the same two for
end-users, and two for experts.
The two end-user interfaces are Browse mode, an alphabetical list of index
terms, and Wilsearch mode, a template or form fill-in interface with labeled
fields, such as Subject, Personal Name, and Title, among others (see Appendix
H). The Wilsearch mode also uses operators such as ANY (the Boolean OR
operator), / (to search by subject heading), # and : (two types of truncation).
Wilsonline expert is a command search mode. The subjects in this study were
allowed to use both Browse and Wilsearch modes. They searched an RGA disc
covering the years 10/83 to 4/92.
The results of this study will ultimately need to be replicated with other
search systems, but one can expect some degree of generalizability due to the
overlap in search features with which users must interact, such as: subject,
title, and author searching, truncation, Boolean logic, online authority lists,
highlighting of menu choices, navigation between screens, print options, etc.
De factor standards for information retrieval systems have been developed over
the years, and most systems include these features (Tenopir and Lundeen, 1988,
p.34). The database was searched on an Insight PC, an IBM compatible computer
with an internal CD-ROM drive, color monitor, and Canon BJ 80A bubble jet
printer.
Back to the top"
5.2 TWO BY TWO INDEPENDENT ANOVA
The design of this study follows a standard 2 x 2 ANOVA with independent
samples (Montgomery, 1991). Figure 6 depicts the details in outline form. The
experiment used 62 subjects randomly assigned to four conditions. The interest
in this study is in novice searchers, hence the subjects were drawn from a pool
of freshmen and sophomores since they were less likely to have experience with
a CD-ROM database system. The variable Type of Instructions has two
conditions: Affectively elaborated user-centered and
unelaborated system-centered. The variable Task Complexity also
has two conditions: complex and simple. Thus, Group 1 (n=17)
received user-centered instructions for the simple tasks. Group 2 (n=14)
received system-centered instructions for the simple tasks. Group 3 (n=15)
received user-centered instructions for the complex tasks. Group 4 (n=16)
received system-centered instructions for the complex tasks.
Searchers in each group were given four query statements and the task of
searching the database to find and print a record that fit the topic of the
query (see Appendix A). They were allotted a maximum of 10 minutes per search
task. Every move each searcher made on the keyboard during the searches was
recorded by the transaction log utility of the system software. This record
was later used to compute the scores for several variables as explained below
(e.g., interactivity, move type, search mode, success).
Subjects in all groups were measured for search style measures: such as
interactivity, search mode, move type, reformulations, and help utilization.
These and the other measures are defined below. As well, data were collected
for evaluation outcome measures: satisfaction, frustration or stress,
perceived self-efficacy as a searcher, helpfulness and comprehensibility of
instructions, and knowledge of the instructions.
5.3 DEFINITION OF DEPENDENT MEASURES
The intent of this experiment was to compare affectively elaborated
user-centered instructions with unelaborated system-centered instructions. It
was important therefore to provide for not just one measure, or even one type
of measure, but a variety of measures designed to assess the broad implications
of the difference between these two types of instructions. This is advised in
research design considerations of validity and is generally referred to as
the construct validation approach (Neale and Liebert, 1986, p.47). The
idea is to demonstrate that there is a "nomological net" (Cronbach and Meehl,
1955), or network of interrelationships, between various aspects of the user's
world. Within each type there are several ways of measuring the variable, a
technique known as convergent validity (Neale and Liebert, 1986, p.85). This
is based on the notion that similar measures of the same thing should correlate
highly with each other, that is, should be affected equally by the same
treatment condition. The variables used in this experiment are the
following:
A. Search Style Measures
1. Interactivity or total number of moves
2. Search mode selected: Browse, Wilsearch, Both
3. Number of conceptual and operational moves
4. Number and depth of reformulations of the
wording of the query statement
5. Consulting point-of-use instructions or Help
utilization
B. Outcome and Evaluation Measures
6. Success or accuracy of printed record to
query topic
7. Satisfaction with results
8. Frustration or stress during searching
9. Perceived self-efficacy as a searcher
or self-confidence
10. Helpfulness and comprehensibility
of instructions
11. Knowledge quiz
The operational definition for each measure is given below, along with the
operational procedures for obtaining the data. The rationale for the predicted
difference between elaborated and unelaborated treatments is explained in each
case.
1. Interactivity or total number of moves
This is defined as the number of moves of all types made by a searcher (Fidel,
1991c). It is an indicator of individual search style. A move refers to any
decision that changes the direction or content of a search task. Examples of
moves are: changing the term entered, modifying the search with a Boolean
operator, combining two or more sets, consulting a thesaurus, marking a record
for printing, deciding to start a new search, and so on. Interactivity is
considered a search style measure since individuals differ on the number of
decisions they make during a single search. The reasons given by searchers for
particular moves in searching have been studied, though a comprehensive theory
of decision-making is not yet to be found in the literature. It is of interest
to this study to determine in what way affectively elaborated instructions
influence interactivity. Too little interactivity might result from lack of
knowledge or reluctance to alter decisions and explore the system features.
Affectively elaborated instructions may facilitate interactivity by
counteracting fear and providing encouragement to experiment. However, it
remains to be seen whether it encourages effective interactivity, that is,
whether the extra moves and strategies are workable or are unnecessary and
ultimately unsuccessful. The number of moves (decisions) made by each searcher
was determined by inspecting the online transaction logs.
2. Search mode selected: Browse, Wilsearch, Both
This is defined as the selection of search mode by each searcher within
each query task. In the present case three possibilities may occur in
conformance with the Wilsonline search software. Individuals may select Browse
mode or Wilsearch mode, or both, by switching back and forth between search
modes. In the literature to date, it is not clear what the consequences of
this choice are for other measures such as interactivity, success,
satisfaction, or self-confidence. The findings of this study, as presented in
the next chapter, reveal some of these relations.
3. Number of conceptual and operational moves
This distinction was previously described and used by Fidel (1991c,
p.515). Conceptual moves alter the content of the search, as in
reformulating the query words or in combining two or more terms. For example,
the query words "Find an article on how much dogs get paid for performing in
movies," were reformulated into search statements such as DOGS AND HOLLYWOOD or
ANIMAL RIGHTS or SALARY ANIMAL PERFORMERS. Operational moves do not
alter the search but move it ahead in its process of completion, as in choosing
a particular function key or changing search mode. Move type is considered a
search style measure and is of interest to this study since user-centered
instructions may help decrease the searcher's reluctance to make use of short
cuts or other powerful features of the system such as truncation or Boolean
operators. Move type scores were calculated from the transaction logs.
4. Number and depth of reformulations made of the query wording
The number of reformulations attempted by a searcher is a measure
described and used by Dalrymple (1990). The depth measure was developed for
this study and used for the first time. Presearch reformulations were obtained
from an Information Need Form (see Appendix A) that subjects filled out after
reading the point-of-use instructions and prior to performing the online
searches. The number of pre-search reformulations was obtained from
this form by counting the number of entries written by the searcher for each of
the four search tasks. The depth of pre-search reformulations was
assigned according to the following definition:
depth 1: searchers write words taken directly from the query
without any change;
depth 2: searchers select words directly from the query but modify
either the order of the words or their form;
depth 3: searchers reformulate the original wording by
reformulating the original once, or modifying the original
once by using allowable operators (ANY, /, #, :)
depth 4: same as depth 3, but more than once.
Depth ratings were made by a graduate student in library science who was
unaware of the study's design or purpose, and validated by the investigator.
It is not theoretically clear how number and depth of reformulations should
relate to type of instructions, task complexity, search mode, self-confidence,
or the other variables in this study. For instance, one might expect a
relation between depth of reformulation and number of strategies attempted.
Another possibility might be that the more reformulations are made, the longer
the search takes. The study also shows whether number and depth of
reformulations are related to success and satisfaction. One might expect that
the greater the number and depth of reformulations, the more the success and
the satisfaction. On the other hand, some reformulations may not work at all,
in which case the search takes longer, with perhaps higher frustration scores.
The findings of this study help provide data for these possibilities.
5. Consulting point-of-use instructions or Help utilization
This is defined as the frequency with which a searcher peruses the
written instructions. This involves three calculations. First, the length of
time spent reading the instructions initially, and second, the length of time
spent subsequently while consulting the instructions during the searches. A
third measure is the total number of times the point-of-use instructions are
consulted during the searches. All three sub-measures are search style
variables and are of interest to this study since user-centered instructions
are expected to be more attractive, less threatening, and thence, more usable.
If a searcher used the online context-sensitive Help, it was also noted. Each
searcher's interaction pattern with the written instructions was entered by the
experimenter on a form showing the times when the instructions were perused,
from which the three sub-measures are calculated. It was predicted that users
who received affectively elaborated user-centered instructions would consult
them more frequently than searchers who received unelaborated system-centered
instructions, since they are expected to be more user-friendly. At the same
time it was expected that it would take longer to read the user-centered
instructions since they are wordier.
Several other expectations can be mentioned. Searchers given the complex tasks
might consult the instructions more frequently given the greater difficulty of
the search problem. However, the more the instructions are consulted, the
longer the search takes. Frequently consulting instructions could increase
success and satisfaction, though it might also increase frustration or stress
during the searching.
6. Success or accuracy of printed record
This measure was defined in broad, rather than narrow, terms. A
graduate student in library science who was unaware of the design or purpose of
the study, assessed each record printed by the searcher by comparing it to the
query. A score of 0, 1, or 2, was given according to the following
definition:
2 = the record matches the specification asked for in the query
1= the record does not match the query specification, but one
the subject headings relates to the query topic
0 = the record does not fit definition 2 or 1, or else, no record
was printed by the searcher
It was expected that success scores would be higher for searchers that were
exposed to the affectively elaborated instructions. As well, simple tasks
would have higher success scores than complex. It was expected that the higher
the success, the stronger the satisfaction. Frustration may be related to
success in a simple way: the greater the success, the less the frustration.
Or, the relationship may be more complex: the greater the success, the less
the frustration, unless it was a long search, in which case, more frustration
will be expressed despite the success. Self-confidence as a searcher may
affect success, though other complicating factors may be involved such as task
complexity and depth of reformulation required for success.
7. Satisfaction with results
This is an evaluation score and is defined as the searchers' rating of
their own satisfaction level with the record they selected to print that
fulfills the query task. A 7-point Lickert-type scale was used for the ratings
from "Not at all satisfied" (1) to "Very satisfied" (7) (see Appendix B). The
experimenter held up the page on which the scale appeared and asked: "How
satisfied are you that the article you found contains the information wanted
for the search task?" It was predicted that satisfaction and success would be
positively related, though the size of the relationship is more difficult to
predict. This depends on whether "being satisfied" depends only on the end
product or as well, on how hard it was to obtain it. If satisfaction with
results is due to multiple factors, then the strength of the relation with
success is less than otherwise.
Satisfaction with results may also depend on perceived self-efficacy as a
searcher: the greater the self-confidence, the higher the satisfaction.
Negative correlations are expected between satisfaction and frustration.
8. Frustration or stress during searching
This is defined by the evaluation ratings given by searchers on the
Frustration/Stress Form (see Appendix C). It is a 7-point scale marked from
"None" (1) to "Quite a bit" (7). The experimenter prompted searchers to give
such a rating at three-minute intervals throughout the search. Since the
maximum time allowed per search was 10 minutes, there was a possibility of
obtaining three frustration probes for each search task. However, it was
expected that some subjects would end the search by printing a record before
the maximum allotted time. Hence the number of probes varied for individual
searchers from one to two to three.
The pattern and level of stress during a search is a function of both user
factors and system factors. Searchers exposed to the user-centered affectively
elaborated instructions could be expected to express less intense frustration
scores given that those instructions included more advice and reassurance,
factors which might proactively reduce frustration or stress during the search.
Searchers given complex tasks may express more frustration. The longer a
search takes, the more frustration may be expressed. Frustration is expected
to relate negatively to success and satisfaction, as mentioned above.
9. Perceived self-efficacy as a searcher or self-confidence
This is a measure of searchers' self-perception in terms of their own
beliefs about their ability and confidence in searching (see Appendix D).
Subjects read five statements of search task performance: "I can complete all
4 tasks with success," and similarly with 3 tasks, 2, 1, and 0. For each
statement, subjects chose "Yes" or "No" and then rated the answer in terms of
certainty or doubt (0% certainty to 100%). The more tasks searchers expected
to complete successfully, the greater their self-confidence as searchers. The
test was administered twice; the first time immediately following the task of
writing presearch reformulations for the four tasks; the second time, following
the four searches.
It was expected that searchers with more self-confidence would be more
successful and satisfied, less frustrated, and take less time to perform each
search task. It is not theoretically clear what the relation should be with
interactivity. On the one hand, one can expect more moves and strategies and
greater depth of reformulations, since these outcomes would be less favorable
for searchers with lower self-confidence. On the other hand, inefficient
interactivity is less likely to produce success on the part of searchers whose
self-confidence is not matched by ability. The findings of this study reveal
more about the pattern of dynamic interaction between these variables in the
searcher's world.
10. Helpfulness and comprehensibility of instructions
This is also an evaluation type score and is defined by four rating
scales from "Not at all" to "Very much" (see Appendix E). The scales are
labeled Helpful, Motivating, Understandable, and Clear. The scores on these
four scales were expected to intercorrelate moderately, though not perfectly,
since they represent somewhat different aspects of helpfulness. The
experimenter prompted searchers to give ratings immediately after completion of
the initial reading of the instructions, when a searcher indicated readiness to
begin the search. A second rating probe was made at the end of the session,
following the four searches. The reason for this second probe was to see
whether performing the searches altered the novices' evaluation of the
instructions, as compared to their evaluation immediately after reading them.
Higher helpfulness scores were expected from those searchers who received the
affectively elaborated instructions. Higher helpfulness and comprehensibility
scores might correspond to greater success and satisfaction and greater
efficiency (less time for each search). Searchers who find the instructions
helpful and comprehensible could make more and deeper reformulations, attempt
more strategies, and their knowledge scores might be higher.
11. Knowledge quiz
This score is defined as the number of correct items answered in a
30-item multiple-choice quiz (see Appendix F). The items were developed to
cover the information in the instructions. The quiz was administered after
searchers read the instructions and wrote down their presearch reformulations.
It was expected that searchers exposed to the affectively elaborated
instructions would have higher knowledge scores since these instructions
contained more information and was longer than the unelaborated version.
Knowledge scores were expected to relate to self-confidence and prior
experience with computers and libraries.
5.4 EXPERIMENTAL CONTROLS
5.4.1 Random Assignment
The hallmark of the true experiment, as discussed in textbooks, is the
practice of random assignment (Neale and Liebert, 1986, p.145). This is
crucial because any treatment effects that are found must withstand an internal
threat to validity from selection bias. When a significant difference
is found, it can only be attributed to the treatment condition. The subjects
were randomly assigned to the four conditions (see Fig. 6). Sixty-two freshman
and sophomore subjects enrolled in an introductory psychology course were
chosen.
5.4.2. Personal Information Form
This form was filled out by subjects at the beginning of the session (see
Appendix G). It asked information about age, gender, academic year of study,
and prior experience with online catalogs, CD-ROM databases, periodical indexes
and word processors. Since a randomized design was used to assign subjects to
the four treatment conditions, no differences were expected between the four
groups on any of the categories in this form. At the same time, these personal
characteristics may correlate with other measures though it is not yet
theoretically clear what these are. Greater experience with online catalogs
and word-processors could be a factor in success and interactivity. The
subjects were drawn from the freshman/sophomore population and it was expected
that very few of them would have had any prior experience with CD-ROM
databases.
5.4.3. Pilot Study
A pilot study was run with the entire design and set of measures that indicated
where changes needed to be made in the instructions (comprehensibility and
length) for both treatment conditions and in the labeling and formatting of
rating scales. The quality of the transaction logs was checked and the
feasibility of obtaining some of the data (such as the periodic
frustration/stress ratings) was tried out and evaluated for adequacy. Minor
changes to the materials and the procedures were made to refine the data
collection, and the study hypotheses were refined. Table 7 illustrates a case
history analysis from the pilot study shows how the data were studied to
improve procedures.
5.4.4. ANOVA Design
The factorial design has a built-in control since the main treatment effect is
checked for two levels of operation. In this case, the predicted advantage of
the affectively elaborated user-centered instructions (the main treatment) is
tried out at two different levels of task complexity (the second treatment).
This approach has the capacity to detect interaction effects. This kind of
interaction effect, whenever found, increases our theoretical understanding of
how the effects operate on one another. In contrast to the simple factorial, a
t-test would be used which is less powerful and incapable of detecting
interaction effects.
5.4.5. Construct Validity
As already described above, this experiment uses a diverse set of dependent
measures to assess the generality of the treatment effect within the world of
the searcher. Thus, outcome measures from transaction logs are in a different
modality than satisfaction or comprehensibility ratings by searchers.
Similarly, knowledge scores from the quiz are cognitive variables while
helpfulness ratings of the instructions are affective variables.
5.4.6. Convergent Validity
For each type of dependent variable, a number of different measures are used.
For instance, search style measures are assessed in five different ways
(interactivity, search mode, move type, reformulations, and frequency of
consulting instructions). Search evaluation outcome measures are assessed in
six ways (success, satisfaction, frustration, self-confidence, helpfulness, and
knowledge). There can be more confidence in the generality of the treatment
effect if the direction of the effect is similar in all of these areas of
search behavior.
5.4.7 Experimenter Bias
Two undergraduate experimenters were trained to administer the procedures with
the subjects (see Section 5.6). They were not informed regarding the design,
theory, and hypotheses of the study. They were novice CD-ROM users.
5.5 INDIVIDUAL STYLE: DEPENDENT OR INDEPENDENT VARIABLE?
There has been discussion in the literature on the relation between personality
factors and searching. The suggestion has been made, for instance, that
people's "cognitive styles" may influence both how they do their searching and
how they best learn the skills of searching. To investigate this issue one
would define cognitive style as the independent variable, and measures of
search behavior or performance would be designated as the dependent variables.
Such a study would be considered a mixed design rather than a true experiment
(Neale and Liebert, 1986, p.183). In what is called a "true experiment" all
factors must be treatment conditions to which subjects are randomly assigned,
and as a result, a cause-effect relationship can be inferred between the
independent and dependent variables. Such is the design of the current study,
since it has two treatment conditions (Type of Instructions and Complexity of
Task) and subjects are randomly assigned to the four cells, as explained above.
Thus, Type of Instructions (the independent variable) has a causal relation to
the dependent measures.
If cognitive style were used as an independent variable, one would first have
to administer a cognitive style test to all subjects and then assign subjects
to different cognitive style groups on the basis of their score on this test.
This represents a restriction on randomization since they cannot be randomly
assigned to the treatment conditions. For instance, those subjects who score
high in one cognitive style category of the test would be placed in one group,
while subjects scoring high in another style category would be placed in
another group, and so on. This is known as a classificatory variable
(like gender, college major, etc.). The reason this is not called a true
experiment is that only a correlational interpretation can be given to the
significance tests. For instance, if one finds a significant difference in
say, number of moves, between those who operate in cognitive style A and those
with cognitive style B, one cannot infer a cause-effect relation between
cognitive style and interactivity scores. The threat to internal validity can
only be eliminated through random assignment, as discussed above.
Back to the top"
A second problem with using cognitive style, or other classificatory variables,
in this kind of study is that point-of-use instructions are made available to
all end-users as part of the search facility. Whatever individual differences
exist in cognitive style, gender, age, subject background, or prior experience,
they are not known to the bibliographic instruction librarians or to the
information providers. To make use of this relation one would have to test all
end-users prior to giving them point-of-use or ondisc instructions, and this is
clearly not a feasible procedure. Instead, the relation being investigated
here, namely the influence of affectively elaborated user-centered instructions
on search behavior and outcome, is theorized to operate generally, no matter
what classificatory characteristics subjects may also have. On the other hand,
some classificatory variables were analyzed, namely, search mode selected by
each searcher and perceived self-efficacy as a searcher. Further, the
correlational technique was also used to reveal possible relations. While
these classificatory and correlational variables do not allow cause-effect
conclusions, they are theoretically useful to reveal networks of interactions
in a dynamic field.
5.6 PROCEDURES WITH SUBJECTS
The subjects were volunteers from undergraduate psychology classes at the
University of Hawaii who received bonus points from their instructors for
participating in the study. The following is the set of experimenter
procedures detailing how every subject was treated. The experimenter went
through the list systematically with each subject.
5.6.1. Experimenter Procedures
Table 8 presents the written instructions that were given to two experimenters
as part of their training with pilot subjects.
5.7 TWO LEVELS OF SEARCH TASKS
The purpose of including two levels of tasks in this experiment was to explore
the applicability of the main treatment effect. If affectively elaborated
user-centered instructions are significantly different, one needs to know
whether the facilitative effects of instructions operate on a broad spectrum of
expertise in the searcher's world. To maximize this breadth, eight tasks were
developed to represent two levels of searching competence (see Appendix A).
The simple tasks were designed to require no query transformation or sequenced
decision-making in either Browse or Wilsearch mode. A novice without prior
experience could accomplish the task with success in either mode. However, to
achieve success with the complex task, searchers must make several steps in
query reformulation and must sequence a number of operations appropriately. A
novice could still have success as long as point-of-use instructions are
followed and consulted when needed. The features that should be used in the
complex task, but not in the simple task, are the following:
-- reformulation of original query
-- broadening and narrowing results
-- looking for related terms in Browse mode
Appendix I presents the instructions for the two levels of tasks, both
user-centered and system-centered.
CHAPTER 6
RESULTS AND DISCUSSION
6.1 INTRODUCTION
The prior chapters discussed the status in the literature of the two factors of
central interest in this study, namely, point-of-use instructions and
interactivity style of searchers. A theoretical rationale was developed for
analyzing the text of instructions in terms of speech acts categorized
according to a behavioral taxonomy comprising cognitive and affective speech
acts at three levels of psychological depth. A factorial design was then used
to assess the effect of type of instructions and of task complexity on
interactivity and success.
Sixty-two novice searchers underwent a search session under controlled
conditions which involved performing four search tasks to be completed within
an allotted time period of 10 minutes for each task. Transaction logs were
recorded, and the bibliographic records printed out by the searchers were rated
for success, assessed in terms of whether the record met the specification in
the query task. Throughout the session and during the search, several measures
were obtained, including searcher's satisfaction with the record printed,
frustration or stress during the search, perceived self-efficacy as a searcher,
knowledge of the search software, and searcher's rating of helpfulness and
comprehensibility of the point-of-use instructions. The findings from ANOVA
and correlational analyses are presented in this chapter, first in detail, then
in summary form (Section 6.5).
The results are discussed within a network of eight topics, as follows:
1. Biographical information about subjects
2. Type of Instructions: unelaborated vs. elaborated
3. Helpfulness and comprehensibility of instructions
4. Task Complexity: simple vs. complex
5. Success and satisfaction with results
6. Search style, interactivity, and search mode
7. Frustration or stress during searching
8. Perceived self-efficacy as a searcher
6.2 DETAILED ANALYSES OF RESULTS
6.2.1 Biographical Information About Subjects
The average age of the 62 subjects is 19 years, with a range from 18 to 27
years. On the average, these individuals report having used an online catalog
20 times and a word processor around 35 times. However, the average use of a
CD-ROM database was reported to be only 4 times. Almost half of the subjects
(45%) reported having never used a CD-ROM database, and most of the other
subjects (40%) had used it between 1 to 5 times. There is no doubt therefore
that this is a population of novices. In fact, 49 of the subjects (79%) chose
the "novice" category to rate their own skill in using CD-ROM databases.
Fifty-six of the subjects (91%) were either freshmen or sophomores. Females
out-numbered males 66% to 34%. Less than half (44%) stated they have used a
print periodical index like the Readers' Guide to Periodical
Literature.
In general there were no significant relationships between biographical factors
and search style or success. The relationships that were found to be
significant, or with a strong trend, are as follows:
1) Females were more successful than males in finding relevant articles in
the database (1.2 vs. 0.9; t=1.3, p=.18). The maximum on the success scale is
2. (The success scores were given by the experimenter on the basis of an
objective assessment, as indicated previously (0=the record printed by the
searcher was unrelated to the query task, or else no record was printed within
the maximum of 10 minutes; 1=the printed record did not answer the specific
query but had subject headings which were related; 2=the printed record
answered the query.) Though the difference between males and females was not
significant, there was a strong trend for women to perform slightly better than
men. For instance, 41% of women (17) had perfect success scores of 2, while
only 26% of the men (5) had perfect success scores. The women in this group
saw themselves as more experienced in wordprocessing, for instance, while 15%
of the women categorized themselves as novices in wordprocessing, 33% of the
men did so.
Since, self-perception is more important a factor than actual ability (Bandura,
1986, p.420), it is possible that this differential perception contributed to
the higher success scores for the women. This explanation is only
hypothetical, since the success difference between males and females did not
reach statistical significance. This is also shown by the fact that women had
slightly lower knowledge scores than men (17 vs. 19). Borgman (1991, p.167)
found no significant gender differences in studies of online search skills
using two different interfaces. Future research will have to determine
whether there is indeed a gender difference in searching.
2) Females gave more extreme helpfulness ratings for the instructions than
males (39.4 vs. 34.8; t=2.0**, p<.05). The maximum on this scale is 56 (see
Appendix E).
3) Females expressed more intense feelings of frustration or stress during
the search (4.0 vs. 2.9; t=2.8**, p<.05). The maximum on this scale is 7
(see Appendix C). The correlation between gender and frustration ratings is
.36 (p<.05), which means that about 13% of frustration during searching is
due to gender and 87% of frustration is due to other factors.
4) The more experience subjects reported in using word processors, the more
they were successful on the search tasks: r=.38, p<.05. This means that
14% of each individual's success score can be attributed to prior experience
with word processors (computers, keyboards, commands, etc.).
5) Subjects (N=24) who reported having used a print periodical index, like
the Readers' Guide to Periodical Literature, obtained higher success
scores for all tasks (1.4), than subjects (N=34) who did not report such
experience (1.0, F=4.2**, p<.05). The correlation between success and prior
experience with print periodical indexes was r=.30, p<.05, thus, 9% of each
individual's success score can be attributed to experience with indexes.
6.2.2 Type of Instructions: Unelaborated vs. Elaborated
The effect of type of instructions on success yielded an F of zero and
an r of zero, indicating no relationship. As will be seen below, task
complexity was a more important factor for success than type of point-of-use
instructions. Nevertheless, here too, there are interactions with specific
tasks. For instance, for task position 2, there was a significant difference
between the two types of instructions: searchers who received the affectively
elaborated instructions indicated higher satisfaction scores than
subjects exposed to the unelaborated instructions (4.7 vs. 3.5, F=4.0**,
p>.05). However, there was no significant difference in success on
task 2 for the two types of instructions, either for the simple or the complex
tasks.
The effect of type of instructions on interactivity style of novices may
be gauged from Table 9 which shows the number of moves of each type in
relation to point-of-use instructions. Only significant contrasts are
presented (missing contrasts are not significant). One can see a general trend
here indicating that the affectively elaborated instructions occasion a
significantly greater number of moves. However, while this trend is true of
all interactivity measures, only some reach significance levels of p<.05.
How does search mode influence success for the two types of
instructions? Table 10 shows such an effect for tasks 1 and 2, but not for
tasks 3 and 4, which are omitted since they are not significant. Searchers who
chose the Browse mode for task position 1, and received the affectively
elaborated instructions, had significantly greater success than those who
selected Wilsearch. With the unelaborated instructions for task position 2,
those who selected the Wilsearch mode were more successful than searchers who
attempted Both modes. These results need to be further explored in future
research to determine which task characteristics interact with instructions and
search mode to yield higher or lower success for novices.
Since search mode preference appears to be an important style factor, it might
be worthwhile to see if type of instructions and task complexity influences
this preference. Table 11 shows the percent of subjects who chose
search modes by type of instructions (five transaction logs were unavailable
due to mechanical failure). It is apparent that two-thirds of these novice
searchers preferred the Wilsearch mode exclusively, irrespective of instruction
type and task complexity. The other third of the subjects split about evenly
between choosing Browse mode only and choosing Both search modes.
The pattern of results reported here with type of instructions indicates that
this treatment needs to be strengthened. Prior to collecting the data, it was
believed that adding affective elaborations to the point-of-use instructions is
all that is needed to influence the searcher. This assumption was true for
some variables (like helpfulness (Table 12) and satisfaction) but not for
others (like success and knowledge scores). Perceived self-efficacy, task
complexity, task characteristics (or query content), and search mode selected
have all proven to be stronger factors than type of instructions. In future
studies it might be advisable to strengthen the treatment effect due to type of
instructions by finding ways to encourage searchers to consult the instructions
more extensively when they have a problem. To the extent that instructions are
not consulted, the treatment effect of affective elaborations has no chance of
influencing searchers.
The type of affective elaborations may also be important and needs to be
explored in future studies. It was believed that changing the A/C ratio from
.9 to 2.3 would be enough to influence searchers. However, it may be that the
elaborations need to be more specifically responsive to the actual problems
searchers encounter with specific tasks. This was attempted on an intuitive
basis, as explained in Chapter 4, but a more systematic approach may be
necessary to increase the strength of the treatment effect. The fact that
helpfulness and comprehensibility scores were significantly more favorable for
affectively elaborated instructions, despite their greater length, indicates
that searchers are positively affected by instructions that give them
orientation, advice, and reassurance (see Table 12).
6.2.3 Helpfulness and Comprehensibility of Instructions
The point-of-use instructions for the CD-ROM searches were rated by users
immediately after reading them initially, just prior to the first search task,
and a second time after completion of the fourth search. Four scales were
given ranging from 1 (not at all helpful) to 7 (very helpful). The four scales
were helpful, motivating, understandable, and clear
(see Appendix E). The rationale for using several scales to assess helpfulness
ratings is to increase reliability and validity. In general, the four scales
were significantly intercorrelated, ranging in strength from a low of r=.30
(p<.05) to a high of .90, p<.05), with a mean around .50, p<.05).
Though significant, these correlations are moderate in strength indicating that
they measure a variety of helpfulness concerns in users. It makes sense,
therefore, to combine them in one total measure of helpfulness to see what
search parameters affect it.
The overall mean for helpfulness of instructions after initial reading was 19.7
(of a maximum of 28.0); this fell somewhat by the end of the fourth search to
18.1 (t=2.8, p<.05). The correlation between them is significant (r=.53,
p<.05), though moderate in strength. This means that only 28% of the
helpfulness scores at the end of the four search tasks is due to rated
helpfulness before the first session. Thus, 72% of the helpfulness scores at
the end was due to the users' experiences during the four searches. In other
words, as the search process cumulates, users continue to be influenced or
guided by the point-of-use instructions.
Helpfulness ratings were also analyzed by the ANOVA method to determine the
effect of type of instructions and task complexity. The affectively
elaborated instructions received significantly better helpfulness ratings
than the unelaborated instructions:
1) for the first administration of the helpfulness ratings
(4 scales, maximum=28), 20.1 vs. 18.4 (F=5.1**, p<.05);
2) for the later ratings, 19.2 vs. 17.0 (F=3.4**, p=.07);
3) for the two combined (twice four scales, maximum=56),
40.1 vs. 35.4 (F=5.5**, p<.05).
This influence of the affective elaborations was nearly uniform for the four
varieties of helpfulness scales, as shown by Table 12 for helpfulness ratings
at the first administration (1 scale, maximum=7) (see Appendix E).
Table 12 demonstrates that subjects gave significantly higher helpfulness and
comprehensibility ratings for the affectively elaborated instructions. This
confirms the treatment manipulation for instructions with the helpfulness
variable. It appears that the elaborations that were added to the original
unelaborated instructions succeeded in addressing some of the affective
concerns users have with point-of-use instructions.
6.2.4 Task Complexity: Simple vs. Complex
There were eight search tasks, four were defined as simple, and the others as
complex. Search tasks that could be performed by using words extracted
directly from the query statement were considered simple. Tasks that required
reformulating the query statement were considered complex. Search
reformulation involves changing the order or form of words in the query
statement, using synonyms, truncating, and recoding, as explained above in the
Procedures. Table 13 presents the search query task statements.
The complex searches took longer than the simpler searches. A search task was
defined as complex if it required reformulating the query through expansion,
contraction, or Boolean combination. Table 14 shows how long, on the average,
each search task took to complete (in minutes). The four simple tasks took on
the average 5.6 minutes to complete, while the complex tasks took 7.1 minutes.
This time difference is significant beyond the 5% level (p<.05). With some
searches, the time difference was not as great (tasks 1 and 3) as in others
(tasks 2 and 4) (see Table 13 for search tasks). A possible explanation for
these task characteristics may be that these novices used reformulations when
it was not desirable, and so spent extra time in unnecessary steps. For
instance, in the simple task 1, success can be obtained by entering "teaching
teens to marry smart" in the title field in Wilsearch mode. However, subjects
did not use the title field, and instead attempted a subject search under
"teenage marriage," which yields an impractically large set. Subjects who used
Browse mode ran into the same trouble. Thus, the simple task 1 took about as
long as the complex task 1.
The subjects were allowed 10 minutes and they decided themselves when they
wished to end the search. Task position was a strong factor in reducing search
length as indicated by Table 15 which combines simple and complex tasks. This
shows that gradually fewer and fewer subjects took the maximum 10 minutes to
complete the search tasks from the first (task position 1) to the last (task
position 4). This points to a learning curve effect: subjects become more
efficient as they cumulate searches. It could also be that fatigue or boredom
sets in.
The correlational technique was used to explore what factors significantly
influence search time. The total time it takes to complete the four search
tasks correlates significantly with frequency of consulting the instructions
during the search (r=.70, p<.05), with number of reformulations of the query
(r=.38, p<.05), with number of conceptual moves during the search (r=.46,
p<.05), and with number of search strategies initiated by users in Wilsearch
mode (r=.44, p<.05). That is, longer searches involved more of these
elements. Total time correlates negatively with success (r=-.69, p<.05),
and with satisfaction (r=.48, p<.05).
These results reveal a dual dynamic of the searcher's world: the more time
they spend on a search, the more moves and strategies are attempted, but the
less success they obtain and the less satisfaction they report. Short searches
for these novices involved fewer moves and strategies, were more successful,
and were rated more satisfying. Future studies will need to investigate the
details of this complex interaction. The greater success and satisfaction with
shorter searches might reflect the motto of "short and sweet": if the correct
strategy is entered, success and satisfaction are quickly obtained, but if a
series of incorrect searches are attempted, time will be longer but success and
satisfaction remain unfulfilled.
The correlation of total time taken for all searches and expressed frustration
or stress is r=0.2, which is not significant. Inspection of the scattergram in
Figure 7 reveals that for about half of the searchers there appears to be a
correlation (cluster A): the longer the searches take, the greater the
frustration; but the other half of searchers show a different pattern. Half of
them (cluster B) show fast search times but high frustration/stress scores;
the remaining searchers (cluster C) show long search times and low frustration.
A similar dynamic was found between success and satisfaction, discussed below
in section 6.2.5.
The results in Table 16 indicate that the effect of task complexity on moves
depends on the specific characteristics of individual search tasks. For task
position 1, the complexity factor was insignificant, that is, complexity did
not systematically influence interactivity. For task position 1, the simple
and complex tasks were: "Find an article that has 'teaching teens to marry
smart' in the title" and "Find an article on computer games that can be used to
simulate the operation of cars." For task 2, complexity occasioned more search
strategies, but the difference for conceptual and operational moves was
insignificant. For task position 2, the two tasks were: "Find an article that
reviews the movie "Ferngully" about the rainforest" and "Find an article on
legal challenges to drug testing of employees in the transportation
industry."
For task 3, complex, ("Find an article on a computer language called Mind")
complexity occasions fewer operational moves and strategies. One explanation
might be that task 3, simple, ("Find an article that relates aging to the mind
and body issue") occasioned more moves and strategies than the complex task 3,
because of its content characteristics. A possible explanation for this
inflated style of interactivity for task 3, simple, might be that those
searchers who selected Browse mode and entered the term AGING, were unable to
find the needed article on "aging and the mind body issue." They thus
performed more interactively, but not necessarily with success.
For task 4, there is a strong effect of task complexity, showing that it
greatly increases interactivity on all measures. For task position 4, the two
tasks were: "Find a humorous article by a student who went to traffic school"
and "Find an article on how much dogs get paid for performing in movies." For
the four tasks combined, complexity occasions significantly greater
interactivity as measured by conceptual moves and total strategies. Further
research will have to disentangle this complex interaction between task
complexity, task characteristics, and interactivity.
6.2.5 Success and Satisfaction with Results
Many of the factors that influence success and satisfaction are mentioned
throughout this chapter. This section summarizes those relationships and
points out some additional ones. The success scores were given by the
experimenter on the basis of an objective assessment, (0=the record printed by
the searcher was unrelated to the query task, or else no record was printed
within the maximum of 10 minutes; 1=the printed record did not answer the
specific query but had subject headings which were related; 2=the printed
record answered the query). The success scores across the four tasks are
significantly intercorrelated, ranging from r=.53 (p<.05) to r=.78
(p<.05). This means that between 28% and 61% of the success scores across
the four tasks measure the same aspect of success, while the rest of the
variability of success scores measures different aspects of success. In other
words, a search task may be seen as having a general or common component, which
it shares with all search tasks, and a specific, unique component which varies
from task to task.
Back to the top"
Future research will have to attempt to disentangle these two components of
success measures. An instance of a general success factor may be experience
with computer systems. In this experiment there was a significant (r=.30,
p<.05), but moderate (9%) effect of experience with word processors and
overall success. Overall success correlates with total satisfaction r=.61,
p<.05, which indicates that 37% of each searcher's satisfaction score is
attributable to success, and 63% of satisfaction depends on factors other than
success. Future research will have to determine what these other factors are.
Two such user factors might be the searcher's expectations of how long
searching should take and the searcher's reaction to zero hits.
Another relation to success is task complexity (r=.35, p<.05), indicating
that approximately 12% of success depends on whether the task is simple or
complex, with simple tasks yielding higher success than complex tasks. More
extensive analysis reveals that this relation is task specific, as shown in
Table 17, which uses t-test to compare the success scores on simple vs. complex
tasks (the maximum is 2).
The overall success for the four tasks combined is significantly higher for the
simple tasks than for the complex. This is as one would expect from common
sense. However, the results in Table 17 demonstrate that this complexity
effect on success varies from task to task, and in this experiment, it is
significant for task 4 only. Future research will have to identify the
specific components of task complexity that are important for success. It can
be expected that what is referred to here as "task complexity" interacts with
the search software and the wording of the query.
Table 22 contrasts success scores for Browse and Wilsearch modes. The overall
success across the four search tasks ranged from a low of 0.8 to a high of 1.6.
In general, greater interactivity for Wilsearch or Both modes did not translate
into greater success. Table 10 shows how type of instructions influenced
success. Searchers who chose the Browse mode for task position 1, and received
the affectively elaborated instructions, had significantly greater success than
those who selected Wilsearch. With the unelaborated instructions for task
position 2, those who selected the Wilsearch mode were more successful than
searchers who attempted Both modes. These results need to be further explored
in future research to determine which task characteristics interact with
instructions and search mode to yield higher or lower success for novices.
Table 25 shows the relation of success to perceived self-efficacy. With both
types of instructions, the more self-confident searchers had higher success
scores. With complex tasks, self-confident searchers had greater success than
less-confident searchers. Overall satisfaction was also higher for searchers
with greater self-esteem than searchers who doubted their ability to succeed.
Success scores were not related to knowledge scores. The mean knowledge
score was 17 (out of a maximum of 30) and this was unrelated to any of the
other variables, such as helpfulness of instructions, type of instructions, or
task complexity. The low scores may be due to the fact that the knowledge quiz
was administered after the initial reading of the instructions and prior to the
searches. Reading point-of-use instructions for a few minutes did allow them
to increase their knowledge somewhat. A parallel finding is reported by Ensor
(1992) in her study of users' keyword search knowledge. The average knowledge
of library users was around 50% for search mode knowledge similar to the
content of the instructions in the present study (see also Naismith and Stein,
1989; Larson, 1983; Martin and Wyman, 1983; Charles and Clark, 1990).
Future studies may want to compare knowledge scores at various times in the
session, to attempt to track cumulating knowledge as the searcher builds up
experience with the search software and the language of subject headings. In
the present study, knowledge scores were correlated with rated helpfulness and
comprehensibility of instructions (r=.35, p<.05) and with satisfaction
(r=.31, p<.05). Thus, the more subjects find the instructions helpful and
clear, the higher their knowledge scores. The higher the knowledge score, the
more satisfaction is expressed with the search. However, the correlation of
knowledge with success was not significant, suggesting once again that more
complex explanations and theories are needed to adequately understand the
dynamics of the searcher's world. This increased understanding would then
allow creation of better point-of-use instructions, better search software, and
ultimately, more satisfaction and success. A theory of search knowledge is
needed: what it is and how it grows. This theory needs to draw upon findings
in cognitive science, as discussed in Chapter 4 (Section 4.1). It may include
such concepts as:
* stored schemas in the searcher's long term memory (e.g.,
commands and functions);
* librarians' cognitive model of searchers (e.g., what vocabulary
they know and what assumptions they are familiar with);
* searchers' cognitive model of librarians (e.g., the meaning of the
words librarians use like index, periodical, citation, subject
heading);
* searchers' system image (e.g., their cognitive attributions as to
what caused what).
6.2.6 Search Style, Interactivity, and Search Mode
At the beginning of each search, subjects started with the menu screen that
allows them to choose either Browse mode (alphabetical list of subjects) or
Wilsearch mode (form fill-in template) (see Appendix H). About two-thirds of
the subjects chose Wilsearch for all four tasks, though there were slight
variations across the tasks, as indicated by Table 18 which shows the
percent of subjects choosing Browse, Wilsearch, or Both. The Chi Square
statistic for Table 18 is significant (12.6, p<.05), which indicates that
the overall preference for Wilsearch by these novices is a reliable finding.
Shneiderman (1987) observes that "there is a paucity of empirical work on form
fill-in, but a number of design guidelines have emerged from practitioners."
(p.123). He compares interaction styles or search modes, including menu
selection, command language, and form fill-in, and recommends the following
with respect to a search mode like Wilsearch:
With the form fill-in interaction style, the users must understand the field
labels, know the permissible values and the data entry method, and be capable
of responding to error messages. ... This interaction style is most
appropriate for knowledgeable intermittent users or frequent users."
(Shneiderman, 1987, p.59).
When this study was planned, it appeared that the Wilson software offered two
search modes to accomodate occasional novice users through Browse mode, and
more frequent users through Wilsearch mode. The point-of-use instructions (see
Appendix I) offer information and advice on both search modes, their
comparative advantages, and the idea that using both, depending on the search
problem, is often advantageous. However, as Table 18 shows, only 18% of
searchers used both modes. Searchers selected the form fill-in template by a
margin of two to one. We discover here an instance of the clash between the
system-centered and the user-centered perspectives. End users simply redefine
the situation which the interface designer has so painstakingly constructed.
The consequence of this clash is low success and high frustration on the part
of the occasional searcher. The majority of novices in this study chose
Wilsearch mode despite their low understanding of field labels and permissible
values, which led them to fundamental errors that persistently obtained zero
hits (e.g., ANDing too many terms or entering title words on the subject line).
These findings imply that point-of-use instructions need to offer more
orientation and advice about search mode logic.
The effect of choosing search mode on the number of moves can also be explored
by the ANOVA technique. Table 19 shows the results for conceptual
moves. As can be seen from the distribution of means, the three search
modes generally occasioned significantly more conceptual moves from Browse, to
Wilsearch, to Both. This differential interactivity due to search mode is,
however, not necessarily related to higher success, as is shown below.
Table 20 looks at the same comparisons for operational moves. These
excluded arrow moves used to look up and down the alphabetical Browse list, but
included Page Up, Page Down, F8 (cross-references), and Enter. In general,
search mode occasions a greater number of operational moves from Browse, to
Wilsearch, to Both, with the exception of Task position 3 where Browse yielded
the most operational moves, even though the arrow moves while searching up and
down the alphabetical Browse mode list were excluded from general operational
moves. This latter result suggests that there may be task specific issues to
be considered with respect to the influence of search mode on interactivity.
The pattern of these findings is replicated with the number of
strategies attempted by searchers, as shown in Table 21. Once again, the
Browse mode occasioned the smallest number of search strategies, while the
searchers who selected both modes generated the most strategies. As pointed
out previously, two-thirds of these novice searchers used only the Wilsearch
mode for all four search tasks.
The relationship between search mode and success may be explored
through the ANOVA technique, as in Table 22, which compares overall success on
the four tasks by all searchers on the basis of their selection of search mode.
The success scores were given by the experimenter on the basis of an objective
assessment, as indicated previously (0=the record printed by the searcher was
unrelated to the query task, or else no record was printed within the maximum
of 10 minutes; 1=the printed record did not answer the specific query but had
subject headings which were related; 2=the printed record answered the query);
(Br = Browse mode, W = Wilsearch mode, Bo = Both modes).
The overall success ranged from a low of 0.8 to a high of 1.6. Note that each
comparison involves a different number of subjects, reflecting the decision as
to which search mode was chosen. Contrasting the Browse mode with Both
(Br/Bo), the Browse mode yielded significantly higher success for task 2, but
not for the other three tasks. This stands in contrast with the finding
discussed above that the Browse mode yielded significantly fewer conceptual
moves, operational moves, and search strategies attempted. In other words,
greater interactivity for Wilsearch or Both modes did not translate into
greater success.
The fact that success was equivalent for Wilsearch and Browse on most of the
tasks in this study, indicates that these novices had trouble using the right
terminology for both modes. They were caught in what Hjerppe calls "the
fundamental paradox of information retrieval" which is "the need to describe
that which you do not know in order to find it" (Hjerppe in Hildreth, 1989,
p.189). Hildreth discusses the assumption justifying Browse mode:
Conventional browsing facilities assume the searcher has a vocabulary-selection
objective to accomplish near the beginning of a search. They assist in
identifying the correct form of a term and any related terms. (Hildreth, 1989,
p.188)
This assumption of the Browse mode failed the majority of novices in the
present study. The difficulty in using Browse mode is the problem of entering
the controlled vocabulary list under the right word form. For example, is it
under "automobile," "car," or "driving"? The searcher has to figure out which
term the indexer selected for the controlled vocabulary in the alphabetical
Browse list. The designer's intent in constructing Browse mode may be met with
some tasks, but not with others. In the present study, as Table 22 shows, the
simple and complex tasks in position 4 occasioned greater success than
Wilsearch, but this was not the case with the other six tasks (see Table 13 for
the wording of the eight tasks).
The results on search mode presented here need to be further explored in the
future regarding their implications for interface design. The novice searchers
in this study showed a clear pattern of preferences for search mode, two-thirds
preferring Wilsearch to Browse mode. The choice of search mode did not
generally influence success, but it did influence search style. Specifically,
Browse mode occasioned fewer moves and strategies. A complex interaction was
found between search mode, success, search style, and search task
characteristic (or query content). On the whole, search mode (which is an
interface design issue) was as important a factor in searching as type of
instructions.
New theoretical proposals for interface design systems (Gardiner and Christie,
1987) are self-consciously explicit about being "user-dependent" and
"context-dependent." A search software that takes into account search style
and search mode preference may be called user-dependent, and if it takes into
account task characteristics (or query content), the interface system may be
called context-dependent. The issue here is that of deciding in each case what
would be the optimal locus of control and agency in a system, so that searchers
can operate with least constraint in any given situation:
The user may be able to deal increasingly in terms of principles which define
operational constraints for the system within which it (sic) is largely free to
determine how goals and subgoals should be achieved (Gardiner and Christie,
1987, p.308).
Future research will have to explore the question of how search mode (and other
interface design options) interact to influence search style or success and
satisfaction. A study is needed to contrast searchers' behaviors when search
mode is assigned to them (treatment condition) and when they are allowed to
choose (as in this study). In what way does search mode create these search
style differences? One hypothesis to be explored is that ability to
reformulate effectively increases success in form fill-in search modes (like
Wilsearch) but not in thesaurus-based modes (like Browse). The issue of
"cognitive compatibility" between users and interface designs has been named as
an important focus for future directions in the psychology of user-interface
design (Gardiner and Christie, 1987, p.319).
One should also explore the possibility that thesaurus-based search modes (like
Browse) may be more advantageous at initial or introductory phases of novice
learning, while form fill-in search modes (like Wilsearch) may be introduced
later. Hammond (1987) recommends this approach: "Providing a restricted
system during training serves to reduce the blind alleys down which the user
will inevitably stray" (p.172). However, these novices had difficulty entering
the Browse list under the controlled vocabulary term, showing that index
language and reformulation represent major obstacles for novices. According to
Clever and Dillard (1991, p.144), "It is probably not too far wrong to state
that good online searching is 90% command of vocabulary and 10% technical
expertise." Shaw (1993, p.372), using heuristic evaluations of the Wilsondisc
CD-ROM interface, identified the tendency to fill-in the entire Wilsearch
screen as an important "usability" problem even for experienced searchers. The
two types of search modes may require different conceptual analogical models
for searchers to understand and operate, and one operational metaphor may be
more helpful at one time of learning than another, as demonstrated by Rumelhart
and Norman (1981).
6.2.7 Frustration or Stress During Searching
Subjects were asked during the searches to indicate their level of frustration
or stress. A written scale was marked from 1 (least) to 7 (most), and shown to
the subjects by the experimenter who took three probes, once approximately
every three minutes (see Appendix C). Subjects who completed their search task
sooner than the allowable 10 minutes did not receive all three probes. The
effect of task position on frustration is shown in Table 23 (A=first
probe; B=second probe; C=third probe).
These results clearly show that frustration varies from task to task. For
instance, for Task position 3 there is a steady and significant decline in
frustration from the beginning of the search (Probe A) to the end (Probe C).
In contrast to this, Task position 4 shows the opposite and significant effect,
namely an increase in frustration from the beginning to the end of the search
time allocated. For Task positions 1 and 2 there are no significant effects.
Neither does the mean for the four tasks combined yield significant differences
across the three probes. Once again, future research will have to explore the
implications by comparing a greater variety of search queries. Clearly, there
are characteristics of the search query tasks that interact with the system
factors, such as the content and organization of the inverted index.
There was a significant effect of task complexity on frustration. For
probes A and B, the simple tasks occasioned more frustration than the complex
tasks (4.1 vs. 3.0, F=8.4**, p<.05 for probe A; for probe B, 4.4 vs. 3.3,
F=4.4**, p<.05). Searchers did not use the easiest available strategy and
spent extra time in non-workable attempts; as a result they may have
experienced more frustration or stress. For instance, task 2, simple, ("Find
an article that reviews the movie 'Ferngully' about the rainforest") had a
simple solution by entering the word FERNGULLY in either Browse or Wilsearch.
Instead, some novice searchers entered FERNGULLY MOVIE REVIEW# RAINFOREST.
This led to zero hits because too many terms were combined in one search
statement with the AND operator, thus including too many restrictions on the
search. This phenomenon, known as the "conjunction fallacy," has been
demonstrated in cognitive science research by Tversky and Kahneman (1983) in
experiments on whether subjects would use probability theory in reasoning
problems.
Boolean searching with logical operators (AND, OR, NOT) presumes an intuitive
understanding of the probabilities involved. Here, the searchers were
apparently unaware that their strategy of including all of the relevant terms
they could think of in one search statement, would have the consequence of
reducing the probability to zero of finding a record that contains all of those
terms. In this study, the majority of searchers in Wilsearch committed the
conjunction fallacy. Clever and Dillard (1991, pp. 149ff) provide several
examples of failed end-user strategies that are contaminated by the conjunction
fallacy. It is obvious therefore that in the future, affective elaborations of
point-of-use instructions ought to explain this fallacy explicitly. If this
were done, the effect of type of instructions on success and self-confidence
could be strengthened beyond what has been found in this study.
Another consequence of this finding is what it implies about defining tasks as
simple and complex. From the skilled searcher's perspective, a task may be
simple in the sense that a readily available search entry leads to relevant
records, while a more complex task would require reformulation and
re-combination of terms before relevant records are obtained. However, this
distinction may not be salient to searchers with less experience who fail to
see the obvious solution for simple tasks and use reformulations and
re-combinations unnecessarily, leading to less success and satisfaction, more
frustration, and longer search times. They thus artificially make simple
search tasks complex. Here too, the implication for the future is to elaborate
instructions to take this distinction into account.
For frustration probe C, task complexity and type of instructions interacted
significantly in a complex way, as shown by the following tabulation of
frustration scores (maximum score=7):
Unelaborated instructions, simple tasks: 2.6
Unelaborated instructions, complex tasks 4.4
Affectively elaborated instructions, simple tasks 4.4
Affectively elaborated instructions, complex tasks 2.7
The two-way ANOVA for this comparison yields F=6.4**, p<.05. Examining the
pattern, it is clear that task complexity had an opposite effect with the two
types of instructions. Thus, with unelaborated instructions, the complex tasks
occasioned more frustration, but with the affectively elaborated instructions,
the simple tasks occasioned more stress. Future research will have to identify
the mechanisms that are responsible for this dynamic situation. Task
characteristics in interaction with search mode properties and timing of the
probe during search, are no doubt responsible. For instance, with Task
position 2 under probe C, the complex task is more frustrating (5.1 vs. 3.4,
F=7.1**, p<.05). But with Task position 3 under probe A, less stress is
reported for the complex task, for both types of instructions (unelaborated
simple =3.2; unelaborated complex =2.9; affectively elaborated simple = 4.4;
affectively elaborated complex =2.0; F=5.8**, p<.05).
Looking at the overall average frustration across all probes, there is a
significant interaction effect (F=5.8**, p<.05), such that for the
unelaborated instructions the frustration is about the same for simple and
complex tasks (3.5 and 3.6), but for the affectively elaborated instructions
the complex tasks occasion less frustration (2.8 vs. 4.3). The affectively
elaborated instructions appear to have an opposite effect on task complexity
since significantly less stress is expressed for the complex tasks (2.8 vs.
3.6) in comparison to the unelaborated instructions.
The correlation between success and frustration is r = -.10, which is
insignificant and negligible. This is rather surprising since one would expect
less frustration with more success. To investigate this further, the
scattergram was examined (see Figure 8) and it became apparent that there are
four patterns or subgroups with respect to how searchers experienced
frustration or stress. One group (cluster A) showed high success and low
frustration; a second group (cluster B) showed high success and high
frustration; the third group (cluster C) showed low success and low
frustration. The fourth group (cluster D) showed low success and high
frustration, which is the usual expectation. Each group was about one fourth
of the total. This kind of pattern suggests that future studies will have to
construct more specific frustration scales, capable of differentiating between
these three types of search reactions by users.
One of the perspectives reviewed earlier in Section 3.2 (Chapter 3) described
searching as moving through a series of barriers, where at any moment, one may
lose one's way. These moments in the search sequence are crossroads where the
searcher needs guidance or new information (Jacobson, 1991). These moments
mark a cognitive gap (see also Kuhlthau, 1991, p.362), so that
information seeking is "gap bridging." The searcher accomplishes this in a
number of a ways, including:
* thinking up an answer
* asking for help
* looking for useful information
* consulting instructions
* formulating questions and hypotheses.
The new "user-centered design philosophy" (Marchionini, 1992) specifically
tries to build user interfaces that are responsive to these gap bridging
attempts by searchers in jeopardy. This perspective recognizes that searchers
seek the path of least cognitive resistance and perform better on systems that
are "pleasant and less frustrating" (Marchionini, 1992, p.158; see also,
Stewart, 1993, p.349-50).
The findings of this study show that frustration is indeed a dynamic condition
of searching by novices. The source of their frustration may be in self-doubt;
that is, in a mismatch between what searchers believe they can do and what they
need to know to come up with a more workable strategy. In this study,
searchers who said "No, I probably can't complete 3 searches successfully,"
reported higher frustration during searching than those who said "Yes." The
self-doubters also had less success, but only with the complex tasks (see Table
25). Here, some of the dynamic is revealed: self-doubters as searchers can
overcome their affective handicap with simple tasks, but not with tasks that
require judicious reformulations of the query words into subject headings.
Future research will have to determine how this handicap in affective search
skills interacts with cognitive inefficiency to yield lower success.
The view in this dissertation has been that weaknesses in affective search
skills lead to cognitive errors, the two being linked. Perhaps searchers who
doubt their capacity to succeed overlook an obvious follow-up to a failed
strategy, or fail to repeat a strategy with slight modifications (e.g.,
truncating a term). Perhaps their system image is cognitively too weak or
inaccurate so that they lack a working metaphor (or algorithm) by which to
interpret search results and plan adequate modifications.
Back to the top"
6.2.8 Perceived Self-Efficacy and Search Style
The perceived self-efficacy measure was intended as an index of searchers'
self-confidence. It consisted of a scale that asks users to indicate whether
or not they think that they can successfully complete 4 search tasks, 3 search
tasks, 2 search tasks and 1 search task (see Appendix D). Subjects rated "How
certain" they were of their answer (from 0% to 100% certain). The measure was
administered twice, the first time immediately after seeing the four search
queries and writing pre-search reformulations for them. The second time was
after completing the four searches. One of the items on the scale, "I can
compete none of the tasks with success" -- Yes/No," contained a double
negative, and subjects gave conflicting answers that were difficult to
interpret. This item is omitted from the presentation of the results.
There were some differences in the data for first and last administration of
the measure, and these are discussed below; however, t-tests and ANOVA analyses
failed to reveal a significant mean difference between the two sets of scores.
This may mean that doing the four searches did not alter the searchers' overall
self-confidence, but future research will have to explore the specific
conditions under which such changes in perceived self-efficacy take place. The
pattern of self-confidence is shown by Table 24 of the perceived-self
efficacy scores for the first administration of the test.
Subjects appear to follow rational rules of inference in estimating their
search ability or performance. Looking at the Yes answers, self-confidence in
successfully completing search tasks steadily increases from four tasks to only
one: 18 subjects (29%) feel they can complete four search tasks, while 60
subjects (97%) feel they can complete one task. The certainty feelings
steadily become stronger, from 61% for four tasks to 84% for one task. Looking
at the No answers, self-doubt appears to weaken steadily from four tasks to
one: 44 subjects (71%) doubted they could do four tasks, but only 2 subjects
(3%) doubted their ability to complete one search task. The degree of doubt
also decreases from four tasks (74%) to one (50%).
The correlational analysis explored what connections may exist between
perceived-self efficacy and other parameters of the study. In general, the
correlations were significant (p<.05) but moderate in strength. For
instance, self-confidence correlates with overall success (r=.45, p<.05),
with satisfaction of search results (r=.50), with frustration felt during the
searches (r=-.45, p<.05), and with helpfulness ratings of the instructions
(r=.63, p<.05). Thus, the more confident these novice searchers feel, the
more successful they are, the more they find the instructions helpful, the more
satisfied they are, and the less frustrated they feel. It should be repeated
that these correlations, though significant, are moderate, indicating that
between 20% and 40% of the common variance is implicated. In other words, 20%
of the success score of every searcher is due to perceived self-efficacy; or,
25% of the satisfaction score is due to perceived self-confidence.
Type of instructions and task complexity did not correlate significantly with
perceived self-efficacy, for either the first or the second testing. In other
words, subjects who received the affectively elaborated instructions did not
exhibit a different pattern of self-confidence than subjects who received the
unelaborated instructions. Similarly, subjects who received the more complex
search tasks did not show a different pattern of self-confidence than those who
received the simpler tasks.
The ANOVA technique allows us to systematically explore the influence of
perceived self-efficacy throughout the rest of the parameters of this study.
Of interest here is to understand the dynamics of the searcher's world. There
is a difference whether searchers believe they can accomplish the search versus
doubting themselves. This time the analysis is applied to the second
administration of the perceived self-efficacy probe. These data were obtained
following the end of the fourth and last search task so that they incorporate
the cumulative effect of four searches upon users. The present discussion is
limited to the "Can do 3 Tasks" probe. Analysis of "Can do 4, 2, and 1 Tasks,"
respectively, yielded very similar results, but the degree of significance was
somewhat less pronounced. Of course the dynamic of self-confidence for 3 tasks
may be slightly different than for 4 tasks, or the other two. This comparison
is expected to reveal whether there is a dynamic interplay between system
factors such as search mode features and user factors, number of moves made,
frustration, success, and satisfaction, as shown in Table 25.
Inspection of the pattern of results reveals two striking effects, already
discussed with the prior analyses. That is, searchers who express
self-confidence ("Yes, I can do 3 search tasks successfully"), are on the one
hand more successful, more satisfied, consider the instructions more helpful,
and report experiencing less frustration. On the other hand, they make fewer
conceptual moves and strategies, spend 41% less time searching than those who
say "no," and thus are more efficient searchers overall. This implies that if
we intend to improve search skills in novices, we should turn our attention to
the affective variable of building self-confidence in online search instruction
sessions and in written point-of-use instructions and Help facilities. The
affective taxonomy of speech acts (Table 2) which guided the affective
elaborations of the instructions can now be now be revised to include elements
of perceived self-efficacy as a searcher.
Section 2.4.1 (Chapter 2) reviewed Bandura's (1989) concept of perceived
self-efficacy in general behaviors. When we apply this perspective
specifically to searchers, a fuller picture emerges as to what the
searcher's world is like:
(a) Searchers are self-conscious information seekers who control their own
destiny by acting (Sensorimotor) through decisions (Cognitive) that are guided
by motives or goals (Affective).
(b) Beliefs about one's own efficacy as a searcher influence search decisions
and strategies; these beliefs relate closely to such sentiments as self-esteem
as a searcher or self-confidence as a searcher.
(c) The stronger searchers' self-efficacy beliefs, the more they persist in
leads they see. On the other hand, self-doubt causes them to abort their
search attempts prematurely and to settle for mediocre solutions.
(d) Frustration or stress during searching is the result of a reduction or
lowering in perceived coping ability in relation to unsatisfactory search
results. The searcher feels a loss of control due to loss of hope in one's
capacity to come up with a workable strategy.
(e) Beliefs about one's searching ability improve in two ways. One is to
experience successes in search tasks. Success overcomes self-doubt. The
second way self-efficacy perceptions improve is through modeling influences.
These include simulations and examples like those one may find in bibliographic
instruction or in point-of-use instructions and online HELP facilities.
6.2.9 Search Literacy and Role Taking
The finding that self-confidence as a searcher is an important factor that
influences search style and success (see Table 25) opens up a potentially new
and important avenue for training people to become better searchers. At first
glance, the relation between search performance and self-efficacy beliefs may
be viewed as pointing to the general importance of individual differences or of
permanent personality traits in searching for information. This approach has
been recently reviewed by Kamala (1992). The literature in psychology on
personality theory, as reviewed for example by Mischel (1993), reflects a
disagreement among researchers regarding the consistency of individual
differences across social situations. One perspective emphasizes the
measurement of traits such as creativity, self-confidence, gregariousness,
aggressiveness, and so on, seeing them as relatively permanent dispositions of
the person based on development and biology. The other perspective emphasizes
the situational environment and shows that performance and conduct vary from
situation to situation. Thus, though individual differences are recognized,
they are not taken as permanent personality traits, but rather as momentary
adaptive response patterns to specific situational demands for coping.
Since searching involves both cognitive and affective coping mechanisms, the
individual is constantly assuming one role or another in the interactive search
process. Searchers who say to themselves, " I think I can complete this search
task successfully," are taking on the role of a search literate individual
(Mead in Turner, 1990, p.375). In contrast, novices who say, "I doubt that I
can complete this search task successfully," are taking on the role of a
technophobic individual. The remedy is to write point-of-use instructions that
facilitate the individual's identification with a search literate role.
In the context of search behavior, perceived self-efficacy may thus be viewed
as searchers' attempts to cope with an unfamiliar and relatively fearful
situation. Searchers who state that they can probably complete three search
tasks successfully (Table 25) are responding with a positive expectancy to the
search situation, while those who state they probably cannot complete tree
tasks successfully are responding with a negative expectancy. It is not
necessary to assume that negative expectancy in the search situation reflects a
permanent disposition of pessimism or low self-esteem, but rather, it may be a
role they are adopting. These same people may take a positive expectancy role
in other situations. Search literacy requires the acquisition of cognitive and
affective skills, both while interacting with the system or librarians, and
while reading or consulting point-of-use instructions. It may be advantageous
to find ways of modifying searchers' expectancies in the search situation
through training or through point-of-use instructions, so that those who
approach the searching situation with negative expectancies can modify them.
This possibility has in fact been demonstrated in experiments designed to
enhance perceived efficacy and coping through the power of positive thinking
(Powell, 1973, Meichenbaum and Smart, 1971). In general, subjects that are
given self-instructional training perform better than controls who are left on
their own in various performance tasks including school work, dart throwing,
interpersonal relations, and problem solving (Mischel, 1993, pp.460ff):
When faced with failure, helpless children seem to have self-defeating thoughts
that virtually guarantee further failure. This became clear when groups of
helpless and mastery-oriented fifth-graders were instructed "to think out loud"
while solving problems. When children in the two groups began to experience
failure, they soon said very different things to themselves. The helpless
children made statements reflecting their loss or lack of ability, such as "I'm
getting confused" and "I never did have a good memory" (Diener and Dweck, 1978,
p.458). None of the mastery-oriented children made lack-of-ability statements.
Instead, these children seemed to search for a remedy rather than for a cause
for their failure. They gave themselves instructions that could improve their
performance, such as "I should slow down and try to figure this out" and "The
harder it gets, the harder I need to try" (Mischel, 1993, p.435).
This and other types of research with adults (e.g., Seligman, 1990) shows that
people acquire a style of explanation that may be either a form of "learned
helplessness" or of "learned optimism" depending on the type of speech act they
use to account for their success or failure at some task. "Pessimism involves
an explanatory style of seeing bad events as enduring, widespread, and due to
oneself" (Mischel, 1993, p.432). In this view, pessimism and doubting in
searching are learned behaviors that evoke self-defeating verbalizations.
Future research will show whether pessimistic verbalizations during searching
can be modified through role-taking techniques that model positive expectancy
speech acts, leading to more efficient, satisfying, and less frustrating
searching (Table 25).
In the case of searching by novices, it is clear that expectations of one's own
efficacy as a searcher importantly guides and directs one's style and decisions
during searching. As shown by Bandura and Adams (1977), asking people to
predict their ability to do a given act successfully, allows good prediction of
the behavior. There is thus a clear link between self-perceptions of one's
competence and the ability to actually behave competently.
People who view themselves as lacking efficacy for coping with life tasks are
vulnerable to anxiety and may develop avoidance patterns designed to reduce
their fears. ... If you expect to succeed, you behave quite differently than
if you are convinced you will fail. Indeed we sometimes behave in ways that
directly help to confirm our expectations, thus enacting self-fulfilling
prophecies (Mischel, 1993, p.427, 406).
The idea that a novice searcher's self-efficacy beliefs may be modified from a
negative to a positive expectation will need to be put to experimental
verification. Stewart (1993, p.349) also found that feelings of
self-confidence as a searcher are important to success in searching because
anxiety interferes with the learning process, so that anxious novices had less
success than self-confident novices. From the perspective of this
dissertation, optimism in searching is a learnable affective skill that
involves generating positive self-regulatory speech acts throughout the
searching process. Point-of-use instructions and online HELP facilities can
either ignore this aspect of the user's environment, or they can attempt to
deal with it explicitly. Future research will have to discover the specific
speech acts that need to be added in the form of elaborations to the
instructions so that searchers will come to expect probable success rather than
probable failure. One approach might be to identify what verbalizations are
used by novice searchers who operate under a positive role model through
think-aloud protocols, and then to insert these sentences in point-of-use
instructions. The use of self-verbalizations as an aid in the self-control of
behavior has been argued by Vygotsky (1965) and demonstrated experimentally in
a number of studies involving problem-solving in academic settings (e.g., Bloom
and Broder, 1950; Roberts, 1979; Roberts and Tharp, 1980).
6.3 INTEGRATING THE DYNAMICS OF THE SEARCHER'S WORLD
Future research may be helped by organizing the findings and emerging
perspective of this study in a single diagram or model. This is presented in
Figure 9. Note that there are 7 layers of
micro-information structure in the
searcher's environment. Beginning with the outer layer, SEARCH QUERY (layer
1), it is specified that its structure derives from the searcher's INFORMATION
NEED, which itself is grounded in CULTURAL TOPICS. The entry point to this
connection is PRESEARCH REFORMULATION of the query, either with or without help
from librarians or instructional interfaces. Query reformulation takes place
through the language of SUBJECT HEADINGS & INDEXES as they are organized by
disciplines into a THESAURUS or other structured list containing
CROSS-REFERENCES (layer 2).
Layer 3 is made up of the SEARCH SOFTWARE which is designed to aid recovery of
a record through INFORMATION RETRIEVAL TECHNIQUES that require interactive
decision-making regarding selection of SEARCH MODE or other OPTIONS such as
SEARCHABLE FIELDS and various available SHORTCUTS, and also requires the use of
LOGICAL and PROXIMITY OPERATORS, and special features such as TRUNCATION.
Layer 4, as we proceed inward towards the searcher, concerns the TEACHING
METHODS used in the POINT-OF-USE INSTRUCTIONS FOR SEARCH SOFTWARE, or other
ONLINE HELP facilities (including bottom of the screen Menu commands). Note
that as we come to layer 5, we are reaching the limits of the searcher's
boundary as an individual.
The present findings are consistent with the principle that point-of-use and
other instructions on search software need to address directly and explicitly
the layers of concerns that searchers have. Using the taxonomic approach
within a behavioral perspective, three environments of the individual searcher
are addressed in point-of-use instructions: the SENSORIMOTOR (layer 5), the
COGNITIVE (layer 6), and the AFFECTIVE (layer 7). Point-of-use and other
instructions (layer 4) address the AFFECTIVE ENVIRONMENT of the searcher by
giving out ORIENTING INFORMATION, ADVICE, and REASSURANCES that legitimize
errors and bolster self-confidence.
The COGNITIVE ENVIRONMENT of the searcher is addressed in the instructions by
providing definitions and explanations. The SENSORIMOTOR environment also
needs to be addressed by the instructions as novice users need practice in
keyboarding skills and in becoming familiar with screen structures (Menu
commands, highlighted texts, position of recurring information and new
information, and other aspects of screen reading ability). The findings have
shown that search style or INTERACTIVITY (which is the number and type of moves
during a search) (Layer 5), varies with individuals even though the same search
task or problem is involved. Excessive interactivity is inefficient and less
successful, so it may be that instructions should also address this feature of
searchers.
The 7 layers of the searcher's world constantly interact. For novice searchers
of a CD-ROM database, SEARCH MODE (in layer 3) influences success, but the more
searchers have SELF-CONFIDENCE (layer 7), the more they are successful
and SATISFIED, and the less FRUSTRATION they experience. At the same time,
they are less INTERACTIVE, using fewer conceptual MOVES, fewer STRATEGIES, and
spend less time searching. They are thus more efficient even though
they are not more experienced. This implies that if we intend to improve the
search style of novices, we should turn our attention to the problem of
building self-esteem in online search instruction sessions and in written
point-of-use instructions and HELP facilities.
A pernicious problem for novice searchers is the CONJUNCTION FALLACY
(layer 6). Many failures are due to ANDing too many search terms in one search
statement. End-users frequently enter a very specific phrase of several words,
sometimes in sentence form (Maxymuk 1991). For example, one searcher trying to
find "a review of the movie 'Ferngully' about the rainforest," entered
FERNGULLY (and) MOVIES (and) REVIEWS
(and) RAINFOREST (and) ANIMATED
which results in a disappointing announcement on the screen "zero hits." The
searchers are apparently unaware that their strategy of including all of the
relevant words they can think of in one statement, will have the consequence of
reducing to zero the probability of finding a record that contains them all.
This is the conjunction fallacy in operation. This has been thoroughly
researched in cognitive science and the conclusion is that the Boolean AND is
not a natural form of reasoning, and it requires a reasoning process that must
be taught explicitly. This problem reveals that novices over-extend most of
their strategies by adding too many terms, using too many operators, and trying
too many strategies. Shaw (1993, p. 374) found that experienced searchers are
confused by the use of both OR and ANY in Wilsearch mode to accomplish the same
search function. Teaching Boolean logic is therefore not enough, novices
probably need to be taught to think in terms of logical operations before
constructing search statements on their own.
6.3.1 Humanization of Technology
The present conceptualization of the searcher's world is compatible with the
latest trend in cognitive science which involves opening itself up to greater
influence from the social sciences as opposed to behavioral psychology, which
helped technologize cognitive psychology as a new "science of the mind"
(Gardner, 1985). The original cognitive revolution of the 1950s, according to
Bruner (1990, p.2), was going to establish "meaning" as the central concept of
a larger, interdisciplinary, and more technologically humanistic social
super-science. But instead of psychology joining forces with linguistics, law,
history, and philosophy, the schism between cognitive science and behaviorism
became technicalized. "The emphasis began shifting from 'meaning' to
'information,' from the construction of meaning to the processing
of information." (Bruner, 1990, p.4) Information theory and information
processing--computations, messages, buffers, constricted choices, system
syntax, heuristics and algorithms--were allowed to displace meaning and
culture--metaphor, connotation, human purpose, and intention. The pendulum of
science has swung too far to the right:
Some critics, perhaps unkindly, even argue that the new cognitive science, the
child of the revolution, has gained its technical successes at the price of
dehumanizing the very concept of mind it has sought to reestablish in
psychology, and that it has thereby estranged much of psychology from the other
human sciences and the humanities. ... The new reductionism provided an
astonishingly libertarian program for the new cognitive science that was being
born. It was so permissive, indeed, that even the old S-R learning theorists
and associationist students of memory could come right back into the fold of
the cognitive revolution so long as they wrapped their old concepts in the new
terms of information processing." (Bruner, 1990, p.7)
The language of reinforcement, "laundered of its affective taint," was
agreeable to the behaviorists when operationalized as cybernetic feedback
mechanisms (Bruner, 1990, p.7). New attacks in cognitive science were launched
against the concept of agency in the form of simulated mental states and
computational networks. According to this mechanistic perspective, intentional
states (beliefs, desires, moral commitments), ought to be eschewed by cognitive
scientists. Mind thus becomes a mere epiphenomenon.
Against this anti-intentionalism trend stands the collective efforts of Goodman
(1984), Searle (1983), and Gergen (1982), among others, who want to construct
"a mental science around the concept of meaning, and the processes by which
meanings are created and negotiated within a community" (Bruner, 1990, p.11).
In this age of technological focus, a deep transformation in social structure
is taking place in our society, and it is time for cognitive science to rely
less exclusively on psychology, and become more influenced by new developments
in sociology, anthropology, political science, and communications. A call for
such a new orientation is clearly advocated by Bruner:
A cultural psychology, almost by definition, will not be preoccupied with
"behavior" but with "action," its intentionally based counterpart, and more
specifically, with situated action--action situated in a cultural
setting, and in the mutually interacting intentional states of the
participants. ... A culturally-oriented psychology neither dismisses what
people say about their mental states, nor treats their statements only as if
they were predictive indices of overt behavior. What it takes as central,
rather, is that the relationship between action and saying (or experiencing)
is, in the ordinary conduct of life, interpretable. (Bruner, 1990,
p.19)
A reflection of the antithesis drawn by Bruner is the orientation clash
emphasized in this dissertation, between system-centered and user-centered
documentation for human-computer interfaces. The focus of the affective
elaborations was on the meaning of the search environment to novice users (see
Figure 8). The humanization of technology can proceed through investigations
into the meanings constructed by users of technology. The taxonomic approach
promises to describe what meanings users attach to technological operations,
and this will allow us to build a bridge between lack of knowledge or inaction,
and empowerment through information literacy and lifelong learning.
6.4 OUTCOME OF THE EXPERIMENTAL HYPOTHESES
H1. There will be no significant difference between the affectively elaborated
instructions and the unelaborated instructions on interactivity scores.
REJECTED
H2. There will be no significant difference between the affectively elaborated
instructions and the unelaborated instructions on frequency of consulting
instructions. NOT REJECTED
H3. There will be no significant difference between the affectively
elaborated instructions and the unelaborated instructions on success scores.
NOT REJECTED
H4. There will be no significant difference between the affectively
elaborated instructions and the unelaborated instructions on satisfaction
scores. NOT REJECTED
H5. There will be no significant difference between the affectively
elaborated instructions and the unelaborated instructions on frustration
scores. NOT REJECTED
H6. There will be no significant difference between the affectively
elaborated instructions and the unelaborated instructions on perceived
self-efficacy scores. NOT REJECTED
H7. There will be no significant difference between the affectively
elaborated instructions and the unelaborated instructions on helpfulness and
comprehensibility scores. REJECTED
H8. There will be no significant difference between the affectively
elaborated instructions and the unelaborated instructions on knowledge scores.
NOT REJECTED
H9. There will be no significant difference between simple tasks and complex
tasks on success scores. REJECTED
H10. There will be no significant difference between simple and complex tasks
on satisfaction scores. REJECTED
H11. There will be no significant difference between simple and complex tasks
on the time it takes to complete. REJECTED
H12. There will be no significant correlation between success scores for the
four tasks (simple or complex). REJECTED
H13. There will be no significant correlation between success and
satisfaction. REJECTED
H14. There will be no significant correlation between perceived self-efficacy
as a searcher (self-confidence) and success scores. REJECTED
H15. There will be no significant correlation between perceived self-efficacy
as a searcher (self-confidence) and satisfaction. REJECTED
H16. There will be no significant correlation between perceived self-efficacy
as a searcher (self-confidence) and frustration during searching. REJECTED
H17. There will be no significant correlation between helpfulness ratings of
the instructions and satisfaction. REJECTED
6.5 SUMMARY OF RESULTS
In this summary, only significant findings are mentioned. The
statistical documentation for each of these findings was presented in the
detailed description that preceded.
6.5.1 Success
a) Females had higher success scores than males. The difference was about
30%.
b) Longer search times produced results that were less successful and less
satisfying than the shorter searches.
c) Success is influenced by the search mode selected by the user.
d) Task characteristics relating to the content of the query and type of
instruction both have an effect on success, though the effect is not uniform
across the search tasks.
e) Success scores for four task positions are significantly intercorrelated,
indicating they serve to assess overlapping components of search ability. The
correlations however, are moderate in strength, indicating that each task
measured its own unique component of ability.
f) The more experience with word processors, the greater the success, though
the size of the relation is small.
g) Success was higher for the four simple tasks in comparison to the complex.
The difference was about 44%. HYPOTHESIS 9
h) The higher the searcher's perceived self-efficacy, the greater the success.
In other words, self-confidence influences success. HYPOTHESIS 14
i) Success was independent of type of instructions. (See also 5a through 5f,
below). HYPOTHESIS 3
6.5.2 Satisfaction with Results
a) The greater the self-confidence as a searcher (perceived self-efficacy), the
stronger the satisfaction ratings with the results. HYPOTHESIS 15
b) The longer the search, the less the satisfaction with the results.
c) The more the instructions are considered helpful and comprehensible, the
more satisfaction is expressed with the search results. HYPOTHESIS 17
d) Satisfaction with results was higher for the simple tasks than for the
complex. HYPOTHESIS 10
e) Success was positively correlated with satisfaction of results. HYPOTHESIS
13
f) The greater the depth of modifications in the reformulations of the query
words, the greater the satisfaction with the results.
g) The affectively elaborated instructions produced better satisfaction with
results for only two of the eight tasks. HYPOTHESIS 4
6.5.3 Interactivity with the System Software and Search Mode
a) The more conceptual moves made by searchers, the longer they took to
complete the search task.
b) The more strategies attempted in Wilsearch mode, the longer they took to
complete the search task.
c) The more reformulations searchers wrote prior to searching, the longer the
search task took to complete.
d) Interactivity, consisting of number of conceptual and operational moves, and
number of strategies attempted, was influenced by the search mode selected. On
the whole, searchers who selected Wilsearch mode, or both Wilsearch and Browse
modes, had higher interactivity scores than searchers who selected Browse mode
only.
e) Interactivity was higher for searchers who were exposed to the affectively
elaborated instructions in comparison to searchers having the unelaborated
version. The difference was about 40%. HYPOTHESIS 1
f) Self-confident searchers exhibit less interactivity.
g) Some tasks produce more interactivity than other tasks, indicating that
search style is influenced by query content.
h) Searchers in this study selected Wilsearch over Browse by a proportion of 2
to 1. This trend is true for both types of instructions, and both simple and
complex tasks.
i) Success was affected by search mode for some tasks, but not for others.
6.5.4. Task Complexity: Simple vs. Complex
a) The complex tasks took longer to complete than the simple search tasks. The
difference is about 27%. (Simple tasks could be solved without changing the
query wording, while complex tasks required reformulation.) HYPOTHESIS 11
b) Some complex tasks occasion more interactivity than other complex tasks,
once more pointing to the importance of the query content and wording.
c) These novice searchers were less successful with complex tasks than with
simple tasks. HYPOTHESIS 12
d) Searchers' self-confidence was not influenced by task complexity. This may
indicate that they are unable to judge in advance the difficulty of a search
task.
e) Searchers exposed to the unelaborated instructions expressed more
frustration with complex tasks than with simple tasks. In contrast, searchers
exposed to the affectively elaborated instructions expressed less frustration
with the complex tasks than with the simple tasks. HYPOTHESIS 5
f) Satisfaction with results was greater with simple tasks than with complex.
HYPOTHESIS 10
6.5.5 Type of Instructions: Unelaborated and Elaborated
a) Affectively elaborated instructions received stronger helpfulness and
comprehensibility ratings than unelaborated instructions. The difference was
around 14%. HYPOTHESIS 7
b) Type of instructions interacts with success and search mode as follows:
searchers exposed to the affectively elaborated instructions had best success
when they selected Browse mode; searchers exposed to the unelaborated
instructions had equal success in Browse and Wilsearch modes.
c) The affectively elaborated instructions occasioned more interactivity than
the unelaborated instructions. HYPOTHESIS 1
d) Success was not different for the two types of instructions. HYPOTHESIS 3
e) Perceived self-efficacy as a searcher is not related to type of
instructions. HYPOTHESIS 6
f) Task complexity and type of instructions interact with frustration as
follows: for complex tasks, there is less frustration or stress under the
affectively elaborated instructions in comparison to the unelaborated
instructions; in contrast, for simple tasks, the unelaborated instructions
occasion less frustration than the affectively elaborated instructions. This
indicates that searchers' frustration can be influenced by point-of-use
instructions, though account must be taken of query content. Future research
needs to study whether there are types of queries that occasion similar
problems for novice searchers, such as queries that evoke too many synonyms, or
those that are specialized and are accessible by controlled vocabulary.
6.5.6 Helpfulness and Comprehensibility of Instructions
a) Female searchers rated the instructions of both types more helpful and
comprehensible than male searchers.
b) Helpfulness and comprehensibility scores correlated positively with
knowledge scores indicating that the clearer they found the instructions the
better they scored on the quiz. However the relationship was moderate
accounting for about 14% of the total variance between the two variables.
c) Self-confident searchers found the instructions of both types more helpful
and comprehensible.
d) Searchers exposed to the affectively elaborated instructions found their
instructions more helpful and comprehensible than searchers exposed to the
unelaborated instructions. The difference was about 20%. HYPOTHESIS 7
e) Helpfulness and satisfaction ratings are intercorrelated, indicating that
the more searchers found the instructions of both types helpful, the more they
were satisfied with the results.
6.5.7 Frustration or Stress
a) Female searchers expressed stronger frustration or stress than male
searchers. The difference was about 38%.
b) Searchers with more self-confidence expressed less frustration or stress
during searches. HYPOTHESIS 16
c) Degree of expressed frustration varied from task to task.
d) Complex searches were more frustrating than simple searches for searchers
exposed to the unelaborated instructions; in contrast, complex searches were
less frustrating than simple ones for searchers exposed to the affectively
elaborated instructions.
e) The more experience with word processors, the less frustration with the
search tasks.
6.5.8 Perceived Self-Efficacy as a Searcher
a) Self-confidence ratings corresponded to the difficulty of the proposition.
Thus, "I can do 4 search tasks successfully" received fewer endorsements than
"I can do 3," and similarly with 2 and 1. This indicates that searchers
perceive themselves as more or less efficacious on the basis of how many
searches they have to complete.
b) Self-confident searchers are more successful, more satisfied with results,
less frustrated during the search process, and spend less time searching.
HYPOTHESES 14, 15, 16
d) Self-confident searchers find the instructions of both types more helpful
and comprehensible. HYPOTHESIS 6
e) The lower the self-confidence of searchers, the more they interact with the
system (moves and strategies).
The above summary of findings gives a picture of the dynamism of the searcher's
world in which type of instructions, task characteristics, and system software
interact to influence user characteristics such as success, satisfaction,
frustration, interactivity, and prior experience. These user factors are not
to be considered problems of individual differences or personality traits.
Instead, they are parameters of a dynamic environment influencing searchers.
The searcher's world is a continuous modification of states, moment by moment,
and the present findings uncover some of the parameters of this network. As
the dynamics of these searcher states are better understood, it will be
possible to influence the states users experience through the parameters that
can be modified such as content of instructions and search software.
(End of Chapter 6)
Back to the top
Forward to Chapter 7
