LIMITATIONS AND FUTURE RESEARCH ISSUES
7.1 TASK CHARACTERISTICS
One of the limitations of this study relates to the choice of the eight search tasks (Table 12). The results showed several task-specific effects, that is, relations or interactions that were significant for some tasks but not others. To eliminate task-specific effects, future studies will need to use larger samples of tasks. Research is needed to identify task features that are relevant to search strategy and outcome. This issue is related to query reformulation into subject heading language. Semantic issues are involved here since query reformulation depends on content, topic, literature organization, and searcher's knowledge, all of which depend on socio-cultural factors.
In this study, task complexity was defined independently of the topics. In future studies, tasks could be pretested on a different sample of comparable novice searchers to determine which tasks are treated by them as simple and which as complex. We need to gain a better understanding of how end users define simple vs. complex.
7.2 SEARCH TIME
In this study, a 10-minute time limit for each search task was imposed on searchers. Although most subjects ended their search before the time limit, some searchers could have increased their success by being allowed to search longer. Or, longer sessions can be planned which would allow more time per search. One needs to investigate a searcher's decision to end a search. What are the reasons? Ostensibly, searchers end a search when they feel they have enough records on the search topic. Yet in this study, searchers ending the search before the allotted time was up, were only moderately successful, and were not fully satisfied. In future studies subjects could be questioned regarding why they have decided to end a search.
7.3 FRUSTRATION/STRESS SCORES
A third improvement relates to the method of obtaining the frustration scores. As was pointed out in the analyses, the frustration or stress question (see Appendix C) did not make it clear what aspect of the searcher's behavior was the frustrating element. Perhaps the experimenter can add the prompt: "Exactly what do you find frustrating right now?" This would reveal whether frustration scores index similar or different problems.
7.4 SEARCH MODE
Since search mode was found to be an important factor for a number of variables, future studies will have to explore the effects investigated here on different systems and different search software. Searchers in this study typically make dozens of moves and strategies in the course of a one-hour search session. Each move represents the outcome of a decision that has a specific goal and offers the possibility of an error. The system interface or search software is the occasion for these errors even though the source of the error lies in the user's lack of knowledge or inattention. As data and theory accumulate on the details of this human-computer interaction we shall be able to design intelligent interfaces that examine searchers' moves online and can suggest error recovery and strategy modification procedures.
7.5 TYPE OF INSTRUCTIONS
The research design should be modified in order to strengthen the treatment effect relating to type of instructions (see Figure 6). There are a number of ways to attempt this. The knowledge quiz could be given as an open-book quiz just prior to searching, so that the instructions would be used as a reference tool for answering the test questions. This may make the information in the instructions more salient prior to searching by ensuring that they are read more than once. Increasing the ratio of affective to cognitive speech acts (A/C) may also strengthen the treatment effect (see Table 6). This could be compared at two or more values to determine an optimal ratio for novices.
In future studies, the kind of elaborations added to the text should be more specifically oriented to those aspects of the searcher's world that are known to be important. For instance, since perceived self-efficacy as a searcher was found to be independent of type of instructions, speech acts specifically aimed at building self-confidence may be added to explore further whether this aspect can be influenced by point-of-use instructions. Longitudinal research on searchers might reveal the etiology of self-confidence as a searcher. To what extent does it depend on general personality factors (self-confidence) and on factors specific to the search setting (e.g., experience, cumulative success history, technophobia, etc.)? Do point-of-use instructions have short term and long term effects? Is there a difference between point-of-use instructions and online HELP facilities in relation to perceived self-efficacy? Is there an interaction between self-confidence and the search interface?
One of the limitations of this study is that the sensorimotor domain was not investigated, though it is recognized that it is an important element that instructions must deal with, along with the affective and cognitive. Future research should address all three domains simultaneously by writing elaborations specifically dealing with sensorimotor issues. A sensorimotor taxonomy will have to be constructed to go along with the affective (see Table 2) and the cognitive (see Table 3). The A/C ratio will thus be joined by A/S and C/S ratios as measures of elaborated texts.
Since user-centered instructions use relational language (see Chapter 4), it is possible that the affective elaborations permanently influence the attitude and perspective of end users who are exposed to them. This possibility is of interest to those who see information seeking as a lifelong learning activity. In what way are the user's aptitude and personality influenced by this new relational language? How does the effect generalize across other information problems such as computerphobia and technophobia? It might be possible to devise experiments that attempt to manipulate searchers' expectations and self-efficacy perceptions as treatment effects (independent variables), in which case, learning and retention can be designated as dependent measures.
When considering the helpfulness of user-centered instructions, one needs to investigate the parameters of this effect. How does it vary across different sub-groups of end-users (novice, advanced, expert, or ethnic and cultural sub-groups)? Does the affective component change in content for experienced end-users? How does the effect vary for different databases (or modes of access) and subject area (thesaurus structure)? How does the affective component vary across multimedia channels? For instance, do graphics, animation, and sound function as affective components of instructions? Are there differences between written information sheets and online help facilities in terms of the content of the affective component that is most facilitative?
We also need to investigate how successful bibliographic instruction librarians can be in writing user-centered instructions. What skills are needed? How can they be acquired? How much wordiness is too much or too little? How does choice of system change the instructions that are needed? What kind of affective speech acts are needed for different information problems? What are the ethics of writing 'perlocutionary' speech acts that attempt to influence searchers' attitudes, self-esteem, or frustration level?
7.6 FACTORIAL INVESTIGATIONS OF THE SEARCHER'S WORLD
This study provides a direction and a strategy for a research program that would investigate the psychodynamic nature of the searcher's world through a series of studies. The data of this study show what variables and processes may be isolated in future studies. For instance, it would be useful to conduct paper and pencil tests of presearch reformulation, Boolean logic ability, search statement creation, search reformulation, and others. Each variable could be studied independently of the dynamic online search environment, to slow down the process searchers must go through, in order to reveal more of the cognitive processes involved.
Table 1
Summary of Kuhlthau's Six-Stage Model
of the Research Process
STAGES AFFECTIVE COGNITIVE SENSORI- MOTOR APPROPRIATE
TASK
1. Initiation Uncertainty General/ Vague Seeking Recognize need
background
information
2. Selection Optimism Scheduling/ Conference Identify
Planning with others
3. Confusion/ Becoming Seeking Investigate
Exploration Frustration informed about relevant
topic information
4. Clarity Narrowed focus Selecting ideas Formulate
Formulation
5. Collection Sense of Defining & Making notes Gather
direction & supporting focus of relevant
confidence info
6. Relief/Satis- Clearer/ More Personalized Complete
Presentation faction or focused synthesis of
Disap- topic
pointment
Table 2
Taxonomy of Affective Instructional Speech Acts in User Documentation
Level Function Cate- Users' Affective Writer's Instructional
gory Behaviors Speech Acts
a Feeling enthusiasm vs. Affirming final outcome
3 REASSURING b displeasure Feeling as successful
(to overcome c empowered vs. helpless Reinforcing user's
resistance d e Showing acceptance vs. perceived self-efficacy
and resistance Having hope Affirming principle that
encourage vs. dread Satisfied users are never at fault
acceptance) vs. dissatisfied Mitigating appearance
of loss Pointing out
value of a result
2 ADVISING a Feeling trust vs. fear Rank ordering options or
(to b of failure strategies Sharing
counteract c d Experiencing fun vs. convenient tips &
anxiety and tedium Feeling information Giving
build confident vs. anxious feedback (what will
positive Experiencing clarity happen if...)
attitudes) vs. confusion Identifying something on
a diagram or giving an
example
Table 2 (Continued)
Taxonomy of Affective Instructional Speech Acts in User Documentation
Level Function Cate- Users' Affective Writer's Instructional
gory Behaviors Speech Acts
a Being patient vs. Telling how long it
1 ORIENTING b impatient Feeling takes (secs. or mins.)
(to reduce c anticipation vs. Identifying common
anger and d frustration Being errors Telling which
maintain e f cautious vs. angry are non- reversible
reality g Having realistic expec- actions Telling what is
check) h tations vs. being excluded when there is
disappointed Feeling doubt Calling
guided vs. lost Being attention to groupings
thankful vs. or sets Showing concern
complaining Being for technical difficulty
realistic vs. Telling what is
disappointed Feeling reasonable to expect
being taken care of vs. Telling where something
being on one's own needed can be found such
as sub-titles
Table 3
Taxonomy of Cognitive Instructional Speech Acts in User Documentation
Level Function Category Users' Cognitive Writer's
Behaviors Instructional
Speech Acts
a b Knowing where in the Giving
2 DEFINING c d text to find info. descriptions,
OPERATIONS (to on specifics vs. Not (incl. titles,
enable knowing headings, labels)
operation of Understanding which Making up a
system) operations belong context for some
together vs. Not operation (incl.,
understanding establishing
Being able to inter-connections)
predict what leads Defining the
to what vs. Not function or use of
knowing what leads some system
to what Being feature (commands,
able to plan keys,
strategies vs. Not procedures)."
being able to proceed Giving a list of
what is included
in a procedure or
label
Table 4
Analysis of Unelaborated Instructions:
Affective and Cognitive Speech Acts
Sentence Affective Cognitive
Number
1 2 3 1h ---- ---- 2a 2b 2c 2a
4 5 6 1h 2a ---- ---- 2d 2d
7 8 9 10 ---- 1d 1h 2a 2a ----
11 12 13 14 2a ---- 2d 2c ---- ----
15 16 17 18 2d ---- 1h 2c 2a 2c 2a
19 20 21 22 ---- 1h 2d 2c 2c 2c 2c
23 24 25 26 2d 2d ---- ---- 2c 2c
27 28 29 30 2d 2d 2c 2c 2c 2a 2c
31 ---- 2c 1e 2a ---- 2b
2d 2d 2d 2d
Note:
The number refers to the Level, while the letter refers to the category (see
Tables 2 and 3).
Table 5
Analysis of Affectively Elaborated Instructions:
Affective and Cognitive Speech Acts
Sentence Affective Cognitive Sentence Affective Cognitive
number number
(cont'd)
1 2 1h 1g 1h 2a ---- 2a 47 48 49 50 ---- 2d 2d 2c ---- 2c
3 4 5 1e 2b 3c 2c ---- 51 52 53 54 3a 2d 2b ---- 2c ----
6 7 8 3b 1h 1d ---- ---- 55 56 57 58 2a 3b 1h ---- ---- 2a
9 10 11 12 2d 2c 2b 2a 2c ---- 59 60 61 62 2d 2d 2d 2a 2a 2a 2a
13 14 15 2b 2c 2d ---- ---- 63 64 65 66 2d 2d 2d ---- ---- 2c
16 17 18 ---- 3b 2b ---- 67 68 69 70 ---- 2a 1h ---- 2a 2b
19 20 21 ---- 3d ---- 2c 71 72 73 74 3b 1h 2b 2c ---- 2c
22 23 24 ---- 2b 2b ---- 2c 75 76 77 78 2b ---- 2b 2c ---- ----
25 26 27 3a 2b 1h ---- 2c 2c 79 80 81 2a 2b 2b ---- 2b ----
28 29 30 ---- 2b 2d ---- ---- 2b 2b 2b ---- ----
31 32 33 1h 1h 2b ---- 2a 2c 2b 2b 3b ---- ----
34 35 2b ---- 2b ---- ---- 2b 2b ---- ----
2c 2a 2a ---- ----
---- 2c
---- ----
Table 5 (Continued)
Analysis of Affectively Elaborated Instructions:
Affective and Cognitive Speech Acts
Sentence Affective Cognitive Sentence Affective Cognitive
number number
(cont'd)
36 37 38 2c 2b 3d 3d ---- 2c 82 83 84 85 2b 3e 2b ---- ----
39 40 1h ---- ---- 2a 86 2d 2b ---- 2c ----
41 42 43 ---- 2d 2d 2c 2c ---- 87 88 89 90 2d 3b 3e ---- ----
44 45 46 1b 2a 3d ---- ---- 91 3a 3a ---- ----
---- ----
Note:
The number refers to the Level, while the letter refers to the category (see
Tables 2 and 3).
Table 6
Quantitative Comparisons Between Unelaborated and Elaborated Versions
Comparisons Unelab-ora Elaborated
ted
Total number of speech acts Number of cognitive 31 25 22 91 35 82 60
speech acts Number of affective speech acts Total ---- 2.4 14 51
speech acts added Average affective speech acts ---- 7 17 0 35
added per cognitive Number of affective speech 15 0 0 14 86 17
acts by levels: 0 25 0 91 1385 6
Level 1 Level 2 7 40 0 60 10 2.3
Level 3 Number of cognitive speech acts by levels: 31 403
Level 1 Level 2 3 ----
Level 3 Number of affective and cognitive speech ---- .9
acts by levels:
Level 1 Level 2
Level 3 Number of sentences Number of words Number
of pages Number of affective speech acts added
Number of cognitive speech acts added A/C
ratio: Total Affective/Cognitive
Table 7
User Strategies: Case History Analysis
REPRESENTATIVE EXPERIMENTAL CONDITION: SUBJECT
Simple Tasks/Affectively Elaborated
Instructions
PSYCHO- * never used CD-ROM, used online catalog a few times *
LOGICAL had used Readers' Guide to Periodical Literature in print *
PROFILE read instructions 7', then several times during search for a
few seconds each time * 19/30 on Knowledge Quiz * rated
AE Instructions "clear" (6) and "motivating" (6) *
Perceived Self-Efficacy, could do 2 of 4 tasks with 50%
certainty * not fully "satisfied" with last search result:
T1=7; T2= 6; T3=7; T4=1 * fully succeeded on 3 (1,2,3) of 4
tasks * "stress" level T1=HI; T2=HI; T3=LO; T4=MID *
after searching, rated instructions "clear" 6;
"understandable" 7; motivating" 6; "helpful" 6 * could do 4
of 4 with 90% certainty, thus gained in Perceived
Self-Efficacy
Note:
See Appendix J for a full analysis of tasks and strategies.Table 8
Experimenter Procedures
Note: Please try to relate to every subject in the identical manner, with the same tone of voice and style. Do not deviate from these written procedures. If subjects ask you for assistance, please inform them that you are not allowed to help, but that they are to rely on the written instructions and their own initiative.
1. Follow instructions on sheet called Wilsondisc Transaction Log Procedures (to set up the computer before subject arrives). Get packet for next subject -- see Subject ID and Group list.
2. Subject fills out the Personal Information Form.
3. Give subject Wilsondisc Instructions and say "You will be given a quiz on this."
4. Keep track of time on Using Instructions Form (Began/Ended).
5. Subject fills out Helpfulness and Comprehensibility Form 1.
6. Subject fills out Information Need Form for all 4 Tasks, and is allowed a maximum of 10 minutes.
REMIND THEM: "Remember you can consult the instructions
at any time."
7. Subject fills out the Perceived Self-Efficacy Form 1.
8. Subject fills out Knowledge Form. NOTE: They may NOT
consult the Instructions for this task.
Note: See Appendix K for the entire set of procedures.
Table 9
Effect of Type of Instruction on Moves
Move Type and Task Contrast (number of Significance F**, p<.05
Position moves) Unelab. vs. Aff.
elab.
Conceptual moves (Task 5.3 vs. 8.1 6.0**, p<.05
1)
Strategies (Task 1) 3.0 vs. 4.7 7.8**, p<.05
Strategies in Wils. 2.4 vs. 4.4 9.2**, p<.05
(Task 1)
Conceptual moves (Task 6.6. vs. 7.1 5.5**, p<.05
2)
Conceptual moves (Task 3.2 vs. 4.8 4.8**, p<.05
3)
Arrow moves (Task 3) 2.0 vs. 2.7 4.8**, p<.05
Table 10
Effect of Search Mode and Instruction Type on Success
Task Instructions BROWSE WILSEARCH BOTH Mean Significance
Posi- Mean (N) Mean (N) (N) F**, p<.05
tion
1 affectively 2.0 (3) 0.7 (16) 1.0 (7) 3.3**, p<.05
elaborated
2 un- 1.3 (7) 1.4 (19). 0.0 (2) 2.5**, p<.05
elaborated
Table 11
Proportion of Subjects Using Search Mode
by Type of Instructions
UNELABORATED AFFECTIVELY ELABORATED
Browse Wilsearch Both Browse Wilsearch Both
6 18 5 4 18 6
21% 62% 17% 14% 64% 21%
25% 54% 21% 13% 71% 16%
Table 12
Helpfulness and Comprehensibility Ratings of Instructions
Immediately After First Reading
Scale Affectively Un-elaborated F test Significance
Elaborated Instructions
Instructions
Helpful 5.5 4.9 5.4** p<.05
Motivating 4.4 3.8 2.0 p=.16
Understandable 5.5 4.8 3.9** p<.05
Clear 5.5 4.8 4.5** p<.05
Table 13
Simple and Complex Search Tasks
Task Simple Search Tasks Complex Search Tasks
Position
1 Find an article that has Find an article on computer games
"teaching teens to marry smart" that can be used to simulate the
in the title. operation of cars.
2 Find an article that reviews Find an article on legal
the movie "Ferngully" about the challenges to drug testing of
rainforest. employees in the transportation
industry.
3 Find an article that relates Find an article on a computer
aging to the mind and body language called "Mind."
issue.
4 Find a humorous article by a Find an article on how much dogs
student who went to traffic get paid for performing in movies.
school.
Table 14
Mean Search Time for Simple and Complex Tasks
Task Position Simple Complex Significance
t**, p<.05
1 7.2 8.1 n.s.
2 4.9 7.5 -3.1**
3 4.8 3.7 n.s.
4 5.4 9.0 -5.1**
Total 22.3 28.3 -3.0**
Mean 5.6 7.1 -3.0**
Table 15
Task Position and Time on Task
Task Position Number of subjects who Percentages
took the max. of 10
minutes
1 28 46%
2 20 33%
3 6 10%
4 1 2%
Table 16
Effect of Task Complexity on Moves
Move Type and Task Contrast (number of Significance F**, p<.05
Position moves) or strategies)
Simple vs Complex
Strategies (Task 2) 2.6 vs. 3.9 4.2**, p<.05
Conceptual moves (Task 3) 4.7 vs. 3.3 3.7**, p<.05
Operational moves (Task 3) 26.5 vs. 12.8 11.1**, p<.05
Op. and Concept. moves 31.1 vs. 16.1 11.3**, p<.05
(Task 3)
Strategies (Task 3) 2.8 vs. 1.9 4.9**, p<.05
Conceptual moves (Task 4) 6.5 vs. 14.0 12.6**, p<.05
Operational moves (Task 4) 19.5 vs. 29.4 7.1**, p<.05
Op. and Concept. moves 26.0 vs. 43.4 12.1**, p<.05
(Task 4)
Strategies in Wilsearch 4.0 vs. 6.6 5.9**, p<.05
(Task 4)
Strategies (Task 4) 4.0 vs. 6.9 8.8**, p<.05
Tot. Concept. moves (all 4 23.0 vs. 32.2 7.5**, p<.05
tasks)
Total Stratgs. in Wils. 10.8 vs. 14.4 4.6**, p<.05
(all 4 tasks)
Total Strategies (all 4 13.4 vs. 16.4 4.0**, p<.05
tasks)
Table 17
Success Scores for Simple and Complex Tasks
Task Position Simple Complex Significance t**,
p<.05
1 0.9 0.8 0.7, n.s.
2 1.4 1.1 1.4, n.s.
3 1.6 1.5 0.5, n.s.
4 1.2 0.3 5.0**, p<.05
Mean 1.3 0.9 2.8**, p<.05
Table 18
Proportion of Subjects Selecting Different Search Modes
Task Position Browse mode % Wilsearch mode % Both modes %
1 21 52 27
2 18 68 14
3 18 68 14
4 11 70 19
Mean 17 65 18
Table 19
Number of Conceptual Moves in Each Search Mode
TASK BROWSE Mean WILSEARCH Mean BOTH Mean SIGNIFICANCE F**,
POSITION (N) (N) (N) p<.05
1 4.1 (12) 6.5 (28) 9.1 (15) 5.1** p<.05
2 3.8 (10) 7.1 (37) 10.5 (8) 4.1** p<.05
3 3.7 (10) 3.3 (38) 7.1 (8) 7.1** p<.05
4 2.8 (6) 11.6 (39) 10.4 (11) 3.0 p=.06
Table 20
Number of Operational Moves in Each Search Mode
TASK BROWSE Mean WILSEARCH Mean BOTH Mean SIGNIFICANCE F**,
POSITION (N) (N) (N) p<.05
1 16.4 (12) 17.8 (28) 27.0 (15) 4.5** p<.05
2 14.6 (10) 17.2 (37) 27.1 (8) 1.5** n.s.
3 31.7 (10) 14.7 (38) 26.6 (8) 5.8** p<.05
4 12.8 (6) 24.9 (39) 30.0 (11) 3.2 p=.05
Table 21
Number of Strategies in Each Search Mode
TASK BROWSE Mean WILSEARCH Mean BOTH Mean SIGNIFICANCE
POSITION (N) (N) (N) F**, p<.05
1 2.6 (12) 3.7 (28) 5.2 (15) 4.8** p<.05
2 1.5 (10) 3.6 (37) 4.6 (8) 4.8** p<.05
3 2.7 (10) 1.7 (38) 4.8 (8) 16.1** p<.05
4 1.8 (6) 5.9 (39) 6.2 (11) 3.6** p<.05
Table 22
Success Contrasted with Search Mode
Contrast Task Overall Success Mean and (number of Significance
Position subjects) F**, p<.05
Br/Bo 1 1.1 (1) vs. 1.2 (16) n.s.
Br/Bo 2 1.2 (34) vs. 0.8 (7) 3.5**, p<.05
Br/Bo 3 1.1 (9) vs. 1.1 (8) n.s.
Br/Bo 4 1.6 (6) vs. 1.2 (11) n.s.
W/Bo 1 1.1 (24) vs. 1.2 (17) n.s.
W/Bo 2 1.2 (34) vs. 0.8 (7) 3.5, p=.08
W/Bo 3 1.1 (34) vs. 1.1 (8) n.s.
Br/W/Bo 2 1.2 (10) vs. 1.2 (34) vs. 0.8 n.s.
(7)
Br/W/Bo 4 1.6 (6) vs. 1.0 (35) vs. 1.2 3.1, p=.06
(11)
Table 23
Effect of Task Position on Frustration Ratings
Task Position Frustration Probes (max. score = Significance F** p<.05
7) A B C
1 3.9 4.0 3.9 0.2, n.s.
2 3.8 4.2 3.9 0.9, n.s.
3 3.5 3.4 2.8 3.1**
4 3.5 4.1 4.1 4.7**
Mean 3.7 3.9 3.5 1.0, n.s.
Table 24
Perceived Self-Efficacy, Self-Confidence, and Doubt
Can do successfully Number of searchers saying How certain are you?
(percent)
Yes No Yes No
4 tasks 18 44 61 74
3 tasks 31 30 66 66
2 tasks 54 8 75 56
1 task 60 2 84 50
Table 25
Perceived Self-Efficacy and Interactivity
User Factors with "Can do Mean for those Mean for those Significance
3 Tasks" saying Yes n=31 saying No n=30 F** p<.05
Number of conceptual moves 3.4 4.9 3.9**
(task 3)
Number of strategies in 1.4 2.4 13.3**
Wilsearch (task 3)
Number of conceptual moves 24.6 32.6 5.2**
(all 4 tasks)
Number of strategies in 11.4 14.9 4.1**
Wilsearch (all 4 tasks)
Number of strategies (all 13.7 16.9 4.3**
4 tasks)
Success with unelaborated 1.7 1.1 5.1**
instructions
Success with affectively 1.8 1.3 3.0, p=.09
elaborated instructions
Success of subjects with 1.7 1.5 1.0, n.s.
the simple tasks (all 4).
Success of subjects with 1.9 1.1 6.8**
the complex tasks (all 4).
Overall success (all tasks 1.3 0.8 19.9**
and conditions) (max.=2)
Satisfaction (all tasks) 17.6 11.9 23.9**
(max.=28)
Table 25 (Continued)
Perceived Self-Efficacy and Interactivity
User Factors with "Can do Mean for those Mean for those Significance
3 Tasks" saying Yes n=31 saying No n=30 F** p<.05
Frustration (all tasks) 3.3 4.0 3.1, p=.08
(max.=7)
Helpfulness ratings of 19.5 16.2 8.4**
instructions (max.=28)
Elapsed Time (all 4 tasks) 5.4 7.6 21.3**
Knowledge scores (max. = 18.0 17.1 0.7, n.s.
30)
Figure 1
Integration of the Affective, Cognitive, and Sensorimotor Domains
Figure 2
Sample Instructions for Using the Printer
Click here for Picture Click here for Picture
Figure 3
Norman's Model of the Human-Computer Interface
Figure 4
Norman's Theory of Action in Relation to the
Affective, Cognitive, and Sensorimotor Domains
Figure 5
Fidel's Typology of Moves in Searching
Task Complexity Type of Instruction
User-centered System-centered
affectively elaborated unelaborated
instructions instructions
Simple Group 1 N=17 Group 2 N=14
Complex Group 3 N=15 Group 4 N=16
Figure 6
Two by Two ANOVA with Four Independent Groups
Figure 7
Scattergram for Search Time and Expressed
Frustration During Searching
Figure 8
Scattergram of Mean Success and Expressed
Frustration or Stress
Figure 9
The Searcher's Dynamic World
Appendix A: Information Need Form:
Complex and Simple Tasks Forms
Complex Tasks Form
Subject I.D.: _______________ Group: ________
Task 1.
Find an article on computer games that can be used to simulate the operation of cars.
You are allowed to consult the written instructions at any time you wish. Please fill in some possible search topics or words that you might start with:
1. ______________________________________________________________
2. ______________________________________________________________
3. ______________________________________________________________
4. ______________________________________________________________
5. ______________________________________________________________
Task 2.
Find an article on legal challenges to drug testing of employees in the transportation industry.
You are allowed to consult the written instructions at any time you wish.
Please fill in some possible search topics or words that you might start with:
1. ______________________________________________________________
2. ______________________________________________________________
3. ______________________________________________________________
4. ______________________________________________________________
5. ______________________________________________________________
IN/C Form (page 2)
Subject I.D.: _______________ Group: ________
Task 3.
Find an article on a computer language called "Mind."
You are allowed to consult the written instructions at any time you wish.
Please fill in some possible search topics or words that you might start with:
1. ______________________________________________________________
2. ______________________________________________________________
3. ______________________________________________________________
4. ______________________________________________________________
5. ______________________________________________________________
Task 4.
Find an article on how much dogs get paid for performing in movies.
You are allowed to consult the written instructions at any time you wish.
Please fill in some possible search topics or words that you might start with:
1. ______________________________________________________________
2. ______________________________________________________________
3. ______________________________________________________________
4. ______________________________________________________________
5. ______________________________________________________________
Simple Tasks Form
IN/S Form
Subject I.D.: _______________ Group: ________
Task 1.
Find an article that has "teaching teens to marry smart" in the title.
You are allowed to consult the written instructions at any time you wish.
Please fill in some possible search topics or words that you might start with:
1. ______________________________________________________________
2. ______________________________________________________________
3. ______________________________________________________________
4. ______________________________________________________________
5. ______________________________________________________________
Task 2.
Find an article that reviews the movie "Ferngully" about the rainforest.
You are allowed to consult the written instructions at any time you wish.
Please fill in some possible search topics or words that you might start with:
1. ______________________________________________________________
2. ______________________________________________________________
3. ______________________________________________________________
4. ______________________________________________________________
5. ______________________________________________________________
IN/S Form (page 2)
Subject I.D.: _______________ Group: ________
Task 3.
Find an article that relates aging to the mind and body issue.
You are allowed to consult the written instructions at any time you wish.
Please fill in some possible search topics or words that you might start with:
1. ______________________________________________________________
2. ______________________________________________________________
3. ______________________________________________________________
4. ______________________________________________________________
5. ______________________________________________________________
Task 4.
Find a humorous article by a student who went to traffic school.
You are allowed to consult the written instructions at any time you wish.
Please fill in some possible search topics or words that you might start with:
1. ______________________________________________________________
2. ______________________________________________________________
3. ______________________________________________________________
4. ______________________________________________________________
5. ______________________________________________________________
Appendix B: Satisfaction Rating
Subject I.D.: _______________ Group: ________
After each search, ASK the subject:
"How satisfied are you that the article you found contains the information wanted for the search task?"
TASK 1: Not at all Very
satisfied ___ ___ ___ ___ ___ ___ ___ satisfied
1 2 3 4 5 6 7
TASK 2: Not at all Very
satisfied ___ ___ ___ ___ ___ ___ ___ satisfied
1 2 3 4 5 6 7
TASK 3: Not at all Very
satisfied ___ ___ ___ ___ ___ ___ ___ satisfied
1 2 3 4 5 6 7
TASK 4: Not at all Very
satisfied ___ ___ ___ ___ ___ ___ ___ satisfied
1 2 3 4 5 6 7
Appendix C: Frustration/Stress
Subject I.D.: _______________ Group: ________
ASK the subject:
What is the level of frustration or stress that you are experiencing with this search task right now?
Record subject's answer and write the time next to it.
_________________
TASK 1: (time a) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
(time b) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
(time c) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
_________________
TASK 2: (time a) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
(time b) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
(time c) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
_________________
TASK 3: (time a) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
(time b) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
(time c) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
FS Form (page 2)
Subject I.D.: _______________ Group: ________
ASK the subject:
What is the level of frustration or stress that you are experiencing with this search task right now?
Record subject's answer and write the time next to it.
TASK 4: (time a) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
(time b) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
(time c) None ___ ___ ___ ___ ___ ___ ___ Quite a bit
1 2 3 4 5 6 7
Appendix D: Perceived Self-Efficacy
Subject I.D.: _______________ Group: ________
Estimate your probable success in these search tasks. Please answer every line.
Write Yes How certain are
or No you? (from 0% to
100% certain)
I can complete all 4 tasks with
success.
I can complete all 3 tasks with
success.
I can complete 2 of the tasks with
success.
I can complete 1 of the tasks with
success.
I can complete none of the tasks with
success.
Do you have any comments or explanations?
_________________________________________________________________
_________________________________________________________________
_________________________________________________________________
_________________________________________________________________
_________________________________________________________________
Appendix E: Helpfulness and Comprehensibility of Instructions
Subject I.D.: _______________ Group: ________
Please rate these instructions as to how helpful and understandable they are to you:
Helpfulness of Instructions
Not at all Very
helpful ____; ____; ____; ____; ____; ____; ____ helpful
1 2 3 4 5 6 7
Not at all Very
motivating ____; ____; ____; ____; ____; ____; ____ motivating
1 2 3 4 5 6 7
Comprehensibility of Instructions
Not at all Very easy to
understandable ____; ____; ____; ____; ____; ____; ____ understand
1 2 3 4 5 6 7
Not at all Very
clear ____; ____; ____; ____; ____; ____; ____ clear
1 2 3 4 5 6 7
Appendix F: Knowledge Quiz
1. The Browse search mode does not allow searching by subject heading.
a. True
b. False
2. What information does a Wilsearch request allow you to type in?
a. subject words; author; journal
b. subject words; author; title; year; language; LC number
c. subject words; personal name; title; journal; year;
organization
3. If you had to find a review of a specific book, which would be preferable or easier to use?
a. Browse mode
b. Wilsearch mode
c. both are equally good for this task
d. neither can be used in this case
4. In Browse mode you can only search a single subject at a time.
a. True
b. False
5. How many subject words can you type in Wilsearch mode?
a. one
b. three
c. four or more
6. If you know the title of a book but not its subject heading or author, which is preferable to use?
a. Browse mode
b. Wilsearch mode
c. both are equally good for this task
d. neither can be used in this case
7. What happens in a Browse search after you enter your subject?
a. you get the first record that fits the subject
b. you get an alphabetical list of subjects
c. one does not type in subjects in Browse mode
8. Suppose that in Wilsearch you type in the following subject words: Data Processing Systems. The computer will show you:
a. records that contain all three words
b. records that contain one of the three words
c. records that contain any of the three words
9. Which of these techniques would be most practical to reformulate your CD-ROM search topic?
a. looking up the word in a dictionary
b. using the list in Wilsearch mode
c. using the list in Browse mode
10. In Browse mode, when there is no match on the screen to the term you typed in:
a. you can type in a new term
b. you can assume that there are no records on that topic
c. both a and b are correct
11. Suppose that in Wilsearch you type in the following:
Subject words: driving
2nd Subject: Hawaii California
The computer will then show records:
a. on driving in Hawaii or California
b. on driving in Hawaii and California
c. on driving in Hawaii but not in California
12. If an * (asterisk) appears before the subject term you have entered in Browse mode:
a. the term you've entered is not a subject in that database
b. no related terms exist for that topic in the database
c. no citations were found for that term
13. The Personal Name line in Wilsearch can be used to look for:
a. a person as an author
b. a person either as an author or as the subject
c. neither a and b are correct
14. You can press [F8] in Browse mode:
a. when there is a number next to the subject entry
b. when there is * (asterisk) next to the subject entry
c. both a and b are correct
15. Suppose you type in the following in Wilsearch :
Subject words: any cholesterol hdl exercise
The computer will show you:
a. records that contain either cholesterol or HDL or exercise
b. records that contain all three topics
c. records on the relation between HDL cholesterol and
exercise
16. The numbers next to each alphabetized subject in Browse mode indicate:
a. how many records were found for each subject
b. how many citations exist for all its related terms
c. both a and b are correct
17. Suppose you type in the following in Wilsearch :
Subject words: /Lopid side effects
The computer will show you:
a. records that contain either Lopid or side effects
b. records on the side effects of Lopid and other drugs
c. records on the side effects of Lopid
18. Which key do you press in order to see related terms to a subject entry in Browse mode?
a. ESC key
b. F8
c. ENTER key, after highlighting bar is on the subject
entry
19. Suppose you type in the following in Wilsearch :
Subject words: rape police:
The computer will show you:
a. records on rape by policemen
b. records that contain the words rape, police, policemen,
policewomen
c. records that contain the words rape or police
20. Which key do you press to examine the first record in Browse mode?
a. ESC key
b. F8
c. ENTER key
21. Suppose you're not sure whether "Jakobovits" is spelled with an "s" or a "z" at the end. Which would be the better format to enter?
a. Jakobovit:
b. Jakobovit#
c. Jakobovits Jakobovitz
22. Which of these techniques would be most practical to reformulate your CD-ROM search topic?
a. looking up the word in Roget's Thesaurus
b. thinking of synonyms
c. looking at records
d. all three (a, b, and c) can be useful
e. a and b are correct
23. When looking at a record in Browse mode, which key do you press when you want to search a new subject in Browse mode?
a. ESC key
b. F8
c. ENTER key
24. Which is generally better when searching for a subject in Wilsearch?
a. using only one grammatical form
b. using different grammatical forms
25. Where do subject headings appear in Browse mode?
a. on the alphabetized subject list
b. on the record found
c. both a and b are correct
26. Which type of search mode allows you to search two or more subjects simultaneously?
a. Browse mode
b. Wilsearch mode
c. Escape mode
27. Which keys do you use for moving the highlighted bar in Browse mode?
a. ENTER key
b. PageUp/PageDown keys
c. arrow keys
d. b and c are correct
28. What is the name of the database you have been searching today?
a. Readers' Guide Abstracts (RGA)
b. CD-ROM Abstracts
c. Social Sciences Index
29. The highlighted bar in Browse mode is used to select:
a. records
b. subject headings
c. screens (previous; next)
30. Which keys do you use to stop displaying entries and resume browsing subjects in Browse mode?
a. ENTER key
b. F4 key
c. ESC key
Appendix G: Personal Information Form
Subject I.D.: _______________ Group: ________
1. Your age: ______ 2. Gender: Male____ Female____
3. UH Student: Fr____ So____ Jr____ Sr____ Grad____
4. Major (actual or contemplated): ______________________________
5. Career goals:__________________________________________________
6. Estimate how many times you have used a library computer catalog (at UH and elsewhere combined):
0 ____; 1-5 ____; 6-15 ____; 16-50 ____; 51-100+ ____
7. How skilled would you rate yourself in using library computer catalogs?
novice ____; beginner ____; skilled ____; expert ____
8. Estimate how many times you have used CD-ROM databases?
0 ____; 1-5 ____; 6-15 ____; 16-50 ____; 51-100+ ____
9. How skilled would you rate yourself in using CD-ROM databases?
novice ____; beginner ____; skilled ____; expert ____
10. Estimate how many times you have used word processors and other computer applications?
0 ____; 1-5 ____; 6-15 ____; 16-50 ____; 51-100+ ____
12. How skilled would you rate yourself in using word processors and other computer applications?
novice ____; beginner ____; skilled ____; expert ____
13. Have you ever used a print periodical index like the Readers' Guide to Periodical Literature?
Yes _____ No _____ Not Sure _____
Appendix I
Database Search Instructions:
Elaborated and Unelaborated Forms
Affectively Elaborated Form
(1)Instructions
(2) Note: There will be a multiple choice quiz on these instructions at the end of the search session today.
(3) SEARCH OPTIONS
(4) There are two options you can choose: Browse search mode and Wilsearch mode. (5) You can start with one, switch to the other, go back to the first, and so on. (6) You cannot injure the program no matter what you do. (7) Just follow your instinct and experiment.
(8) BROWSE SEARCH
(9) Browse search only allows you to search a single subject at a time from an official list on the screen. (10) For example, say you're interested in the subject of "Mind control and adolescents." (11) If you search "Mind control" as a single subject, you will get all the articles that are about Mind control. (12) You can also search the topic of "adolescents" as a separate search. (13) On the other hand, with the Wilsearch mode (to be explained below), you can search two or more subjects simultaneously, such as "Mind control and adolescents," in which case you will get only those articles that deal with mind control among adolescents-- which is a more efficient (smaller) set to look through to find the one article you want.
(14) In a Browse search, after you type in your subject, you get a screen display which shows an alphabetical list of subjects. (15) If there is no subject that matches exactly what you typed in, inspect the list on the screen for other subjects that may relate to your subject. (16) Use Up and Down arrow keys, or PageUp and PageDown keys, to move the highlighted bar up and down the Browse list. (17) Feel free to explore and inspect as much as you want.
(18) If an * (asterisk) appears before an entry on the list, no records were found for that term. (19) However, there is still something very useful you can get out of it. (20) By pressing the [F8] key for any highlighted subject, including those with an * (asterisk), a selection of related terms will be listed. (21) Use the esc (escape) key to return to the Browse list when you want to look at more subjects. (22) To start a new search, just start typing and the computer will take you to the new place on the Browse list. (23) You'll quickly get used to this, don't worry. (24) Watch the bottom of every screen where you're told what keys you can use to move back and forth.
(25) WILSEARCH
(26) By allowing the combination of two or more subject terms, Wilsearch enables you to perform a more complex search. (27) That is, you will be able to search for an article that simultaneously contains two or more subjects you desire, an approach which can save you a lot of time and effort.
(28) Look at the box showing you what the screen looks like in Wilsearch mode.
(29) Wilsearch Screen (30) Explanations
ENTER YOUR LOCAL SEARCH REQUEST SUBJECT (31) Type each aspect of your
WORDS: abortion 2nd SUBJECT: any moral topic on a separate line. (32) If
ethics 3rd SUBJECT: PERSONAL NAME: Mead, using alternative or synonymous
Margaret TITLE WORDS: JOURNAL NAME: terms, type the word ANY followed
ORGANIZATION: DEWEY NUMBER: You do not by all terms. (33) This field can
need to fill out the entire screen. Use be used to look for a person
only those lines appropriate to your either as an author or as the
search request. F1=HELP F2/ESC=QUIT subject. (34) After you have
END=End Input filled in the lines describing
your topic, press END, followed by
ENTER.
(35)
At this screen if you would press END followed by ENTER, the
computer will search for articles written by Margaret Mead on abortion
and either moral or ethics. (36) If you leave out the
author on the Personal Name line, the computer will search for records on
abortion and moral or ethics written by anyone.(37) Use the esc key to move back and forth between Wilsearch and Browse modes, or between viewing a record and starting another search. (38) Don't be discouraged if you find nothing for a search (this is called "zero hits"). (39) It's easy and quick to modify the search until you have just the right combination of words for success.
(40) TIPS FOR YOUR SEARCH
(41) 1. In Wilsearch mode, when there are several synonyms, enter each on the same line preceded by the word "any". (42) Do not separate the terms with commas:
Example: any children youth adolescents
(43) This means that you desire an article that has any one of these three words as its subject. (44) Note that if you leave out the word "any," then the computer will find only those records that have all three words in their subject. (45) Clearly, you wouldn't want that since you're likely to get zero hits that way. (46) If you make a mistake, or change your mind, you can always go back and change the search without any trouble.
(47) 2. If you know the term you are searching is an exact subject heading (from Browse search), enter "/" (slash mark) before the search term. (48) This will restrict retrieval to subject headings and will help to eliminate unrelated articles:
Example: /civil rights
(49) 3. If you wish to expand the spelling of your search term, truncate the term in one of two ways:
# = adds any single letter.
Example: automobile#
[the computer will search for automobile or
automobiles]
(50) This is a nice feature that can increase the likelihood you will get an article that is relevant to your topic.
(51) : = adds unlimited number of letters.
Example: war:
[the computer will search for war, wars, warts,
ward, warble, warrior, etc.]
(52) Obviously, this would give you a lot of articles that you don't want. (53) So you need to figure out which truncation method is best in any particular situation. (54) You'll get the hang of it as you experiment!
(55) UNDERSTANDING YOUR SEARCH RESULTS
(56) author = O'Neill, Onora Children's rights and (57) = title (58) journal children's lives Ethics 98: 445-63 Ap'88 (59)= volume, = SUBJECTS COVERED Children/Civil rights pages, & date(60) You are looking at a diagram of a portion of the computer screen (box). (61) It is displaying part of a record that appears on the screen during your search.. (62) Use the esc (escape) key to return to Browse mode and type in the subject terms that you see on records. (63) Or, you can choose Wilsearch mode, and type in the subject terms you saw on the record (might be a good idea to write them down!).
(64) TO PRINT
(65) When you find an article you want, press [F4] to print the entry.
(66) HELPFUL STRATEGIES IN SEARCHING
(67) 1. Start on the Subject line in Wilsearch mode using words from the search task as well as all the synonyms you can think of. (68) However, if it is a name you are searching, use the Personal Name line.
(69) 2. Each record found is called a "hit." (70) Examine a few of the records looking at the line marked Subjects Covered. (71) You don't need to examine all the records found, especially if it is a large set (say, 20 or more). (72) Write down subject words that you think might be useful to search on.
(73) 3. If you obtain "zero hits" (this will be shown on the screen), then you can modify your search by retyping the words you had typed in earlier. (74) Try different combinations of words you already have. (75) Or, try different words. (76) Or, eliminate some of the words. (77) Then try the modified or new search. (78) Then try again. (79) And so on, until you're satisfied.
(80) 4. If you obtain zero hits again, switch to Browse mode and type in some of the subject words (one at a time). (81) Inspect the Browse list each time. (82) Up and down, and on the next or previous page. (83) One of the entries might fit your topic. (84) Or, use the [F8] key to see related terms, and again, you can press [F8] to see related terms to those, and so on.
(85) The screen will always show you how many records there are for each entry line on the list.
(86) 5. Examine the records. (87) Again look at the Subjects Covered to get more ideas on what words to search on.
(88) 6. In summary: Feel free to go back and forth several times between Browse and Wilsearch modes, and between them and the records. (89) Write down new words you notice and try them until you have the article you're satisfied with. (90) Good luck!! (91) It could be exciting!!
Unelaborated Form
(1) Instructions
(2) Note: There will be a multiple choice quiz on these instructions at the end of the search session today.
(3) SEARCH OPTIONS
(4) BROWSE SEARCH
(5) This is the simplest method of searching. (6) Browse search allows you to search a single subject. (7) If an * appears before the subject term you have entered, no citations were found in the database for that term. (8) However, by pressing [F8] for any subject heading, a selection of related terms that can be searched will be listed. (9) The subject headings that appear can also be used when doing a more complicated search in Wilsearch.
(10) WILSEARCH
(11) By allowing the combination of two or more subject terms, Wilsearch enables you to perform a more complex search.
(12) Wilsearch Screen (13) Explanation
ENTER YOUR LOCAL SEARCH REQUEST SUBJECT (14) Type each aspect of your
WORDS: abortion 2nd SUBJECT: any moral topic on a separate line. (15) If
ethics 3rd SUBJECT: PERSONAL NAME: Mead, using alternative or synonymous
Margaret TITLE WORDS: JOURNAL NAME: terms, type the word ANY followed
ORGANIZATION: DEWEY NUMBER: You do not by all terms. (16) This field can
need to fill out the entire screen. Use be used to look for a person
only those lines appropriate to your either as an author or as the
search request. F1=HELP F2/ESC=QUIT subject. (17) After you have
END=End Input filled in the lines describing
your topic, press END, followed by
ENTER.
(18) TIPS FOR YOUR SEARCH
(19) 1. If there are several synonyms for one aspect of your search, enter each on the same line preceded by the word "any". (20) Do not separate the terms with commas:
Example: any children youth adolescents
(21) 2. If you know the term you are searching is a subject heading (from BROWSE SEARCH), enter "/" preceding the search term. (22) This will restrict retrieval to subject headings and will help to eliminate unrelated articles.
Example: /civil rights
(23) 3. If you wish to expand the spelling of your search term, truncate the term in one of two ways:
# = adds any single letter.
Example: automobile#
[the computer will search for automobile or automobiles]
(24) : = adds unlimited number of letters.
Example: war:
[the computer will search for war, wars, warts,
ward, warble, warrior, etc.]
(25) UNDERSTANDING YOUR SEARCH RESULTS
(26) author = O'Neill, Onora Children's rights and (27) = title (28) journal children's lives Ethics 98: 445-63 Ap'88 (29)= volume, = SUBJECTS COVERED Children/Civil rights pages, & date
(30) TO PRINT
(31) When you find an article you want, press [F4] to print the entry.
Appendix J: User Strategies: Case History Analysis
REPRESENTATIVE EXPERIMENTAL CONDITION: SUBJECT
Simple Tasks/Affectively Elaborated Instructions
PSYCHO- * never used CD-ROM, used online catalog a few times *
LOGICAL had used Readers' Guide to Periodical Literature in print *
PROFILE read instructions 7', then several times during search for a
few seconds each time * 19/30 on Knowledge Quiz * rated
AE Instructions "clear" (6) and "motivating" (6) *
Perceived Self-Efficacy, could do 2 of 4 tasks with 50%
certainty * not fully "satisfied" with last search result:
T1=7; T2= 6; T3=7; T4=1 * fully succeeded on 3 (1,2,3) of 4
tasks * "stress" level T1=HI; T2=HI; T3=LO; T4=MID *
after searching, rated instructions "clear" 6;
"understandable" 7; motivating" 6; "helpful" 6 * could do 4
of 4 with 90% certainty, thus gained in Perceived
Self-Efficacy
TASK 1 [1] Chooses Wilsearch [2] Selects 3 topic terms from the
STRATEGY 1 problem specification and presearch reformulations (Info Need
RESULT Form) [3] Truncates one term for plural element Zero hits
TASK 1 [1] Escapes and chooses New Search [2] Keeps 2 of the
STRATEGY 2 previous terms, but truncates them for all ending variations
RESULT [3] Selects a new term and truncates it for all variations
ANALYSIS 3 hits Views set, then prints #2 It is the correct choice
Time elapsed = 5' 17 moves 6 conceptual decisions 11
operational decisions 2 strategies Successful strategy
truncated for variations, and selected three main topical
elements from the problem specification
TASK 2 [1] Chooses New Search in Wilsearch [2] Selects 3 topic
STRATEGY 1 terms from the problem specification and presearch
RESULT reformulations (Info Need Form) Zero hits
TASK 2 [1] Chooses New Search [2] Keeps 2 of the search terms
STRATEGY 2 [3] Selects a new term from the problem specification Zero
RESULT hits
TASK 2 [1] Chooses New Search [2] Selects all 4 search terms
STRATEGY 3 used so far and truncates each of them for all ending
RESULT variations Zero hits
TASK 2 [1] Chooses New Search [2] Uses same 4 search terms used
STRATEGY 4 in Strategy 3, but uses OR operator to combine 2 of them 2
RESULT hits Views set and prints #2 It is correct choice time
ANALYSIS elapsed = 9' 32 moves 12 conceptual moves 20 operational
moves 4 strategies Satisfaction rating 6
TASK 3 [1] Chooses New Search [2] Selects 3 topic terms from the
STRATEGY 1 problem specification [3] Uses OR to combine 2 of these
RESULT several hits views set and prints #3 time elapsed = 3' 12
ANALYSIS moves 2 conceptual moves 10 operational moves 1 strategy
fully satisfied successful strategy was ANDing 2 of the main
topic terms
TASK 4 [1] Chooses New Search [2] Selects 4 main topic terms
STRATEGY 1 from problem specification Zero hits
RESULT
TASK 4 [1] Chooses New Search [2] Selects same 4 terms, but
STRATEGY 2 truncates 2 using : Zero hits
RESULT
TASK 4 [1] Chooses New Search [2] Selects same 4 terms, but uses
STRATEGY 3 OR to combine 3 of them Zero hits
RESULT
TASK 4 [1] Chooses New Search [2] Selects same 4 terms, but
STRATEGY 4 truncates 1 using : Zero hits
RESULT
TASK 4 [1] Chooses New Search [2] Adds a 5th search term, uses
STRATEGY 5 OR to combine 2 terms Zero hits
RESULT
TASK 4 [1] Chooses New Search [2] Repeats Strategy 2 Zero hits
STRATEGY 6 time elapsed = 10' 45 moves 17 conceptual moves 28
RESULT operational moves 6 strategies not satisfied did not use
ANALYSIS Browse mode to locate TRAFFIC SCHOOLS/ANECDOTES
Appendix
K: Experimenter Procedures
Note: Please try to relate to every subject in the identical manner, with the same tone of voice and style. Do not deviate from these written procedures. If subjects ask you for assistance, please inform them that you are not allowed to help, but that they are to rely on the written instructions and their own initiative.
1. Follow instructions on sheet called Wilsondisc Transaction Log Procedures (to set up the computer before subject arrives). Get packet for next subject -- see Subject ID and Group list.
2. Subject fills out the Personal Information Form.
3. Give subject Wilsondisc Instructions and say "You will be given a quiz on this."
4. Keep track of time on Using Instructions Form (Began/Ended).
5. Subject fills out Helpfulness and Comprehensibility Form 1.
6. Subject fills out Information Need Form for all 4 Tasks, and is allowed a maximum of 10 minutes.
REMIND THEM: "Remember you can consult the instructions
at any time."
7. Subject fills out the Perceived Self-Efficacy Form 1.
8. Subject fills out Knowledge Form. NOTE: They may NOT
consult the Instructions for this task.
9. Subject starts search on Task 1 in the Information Need Form, and is allowed a maximum of 10 minutes..
REMIND THEM: "Remember you can consult the instructions
at any time."
10. Keep track of Task 1 on Using Instructions Form
(Began/Ended)
11. Show subject Sample Frustration/Stress Scale and record answer on FS Form. Do this 3 times: (a) after 1 to 2 mins.; (b) 2 to 3 minutes later; (c) when subject prints article found.
12. Subject prints record found at the end of the search on Task 1.
13. Show subject Sample Satisfaction Scale and record answer for Task 1 on S:Oral Form.
14. Subject starts search on Task 2 in the Information Need Form, and is allowed a maximum of 10 minutes.
REMIND THEM: "Remember you can consult the instructions
at any time."
15. Keep track of Task 2 on Using Instructions Form
(Began/Ended)
16. Show subject Sample Frustration/Stress Scale and record answer for Task 2 on FS Form. Do this 3 times: (a) after 1 to 2 mins.; (b) 2 to 3 minutes later; (c) when subject prints article found
17. Subject prints record found at the end of the search on Task 2.
18. Show subject Sample Satisfaction Scale and record answer for Task 2 on S:Oral form.
19. Subject starts search on Task 3 in the Information Need Form, and is allowed a maximum of 10 minutes.
REMIND THEM: "Remember you can consult the instructions
at any time."
20. Keep track of Task 3 on Using Instructions Form.
(Began/Ended)
21. Show subject Sample Frustration/Stress Scale and record answer for Task 3 on FS Form. Do this 3 times: (a) after 1 to 2 mins.; (b) 2 to 3 minutes later; (c) when subject prints article found
22. Subject prints record found at the end of the search on Task 3.
23. Show subject Sample Satisfaction Scale and record answer for Task 3 on S:Oral form.
24. Subject starts search on Task 4 in the Information Need Form.
REMIND THEM: "Remember you can consult the instructions
at any time."
25. Keep track of Task 4 on Using Instructions Form.
(Began/Ended)
26. Show subject Sample Frustration/Stress Scale and record answer for Task 4 on FS form. Do this 3 times: (a) after 1 to 2 mins.; (b) 2 to 3 minutes later; (c) when subject prints article found
27. Subject prints record found at the end of the search on Task 4.
28. Show subject Sample Satisfaction Scale and record answer for Task 4 on S:Oral form.
29. Subject fills out Form.
30. Subject fills out PSE:2 Form.
31. Thank the subjects when they are leaving and express hope they have learned something useful they can use in their courses or in life.
32. Follow these steps to get ready for the next subject:
(1) Quit the Disc Search Menu
(2) At prompt c:\wils2-4> copy trace.log a: subjXX
(the XX stands for the Subject ID)
(3) At prompt c:\wils2-4> print trace.log
(4) At prompt c:\wils2-4> wils2-4
(puts you back on the Disc Search Menu)
(5) Tear off printout and mark it with Subject ID and Group.
(6) Collect packet for subject, clip together, and place in folder.
33. Go get the next subject by the elevator tables or chairs and start with Step 2 above.
34. When you have time, score the subjects' Answer Sheets in the packets.
Austin, J. L. (1965). How to do things with words. New York: Oxford University Press.
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall.
Bandura, A. (1989). Human agency in social cognitive theory. American Psychologist, 44, 1175-1193.
Bandura, A. & Adams, N. E. (1977). Analysis of self-efficacy theory of behavioral change. Cognitive Therapy and Research, 1, 287-310.
Bandura, A., Reese, L. & Adams, N. E. (1982). Microanalysis of action and fear arousal as a function of differential levels of perceived self-efficacy. Journal of Personality and Social Psychology, 43, 5-21.
Barling, J. & Abel, M. (1983). Self-efficacy beliefs and tennis performance. Cognitive Therapy and Research, 7, 265-272. Belenky, M. F., Clinchy, B. M., Goldberger, N. R. & Tarule, J. M. (1986). Women's ways of knowing: The development of self, voice, and mind. New York: Basic Books.
Belkin, N. J. (1980). Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science, 5, 133-143.
Belkin, N. J. (1984). Cognitive models and information transfer. Social Science Information Studies, 4, 111-130.
Bellardo, T. (1985). An investigation of online searcher traits and their relationship to search outcomes. Journal of the American Society for Information Science, 36, 241-250.
Bloom, B. S., (Ed.). (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York: David McKay.
Bloom, B. S. & Broder, L. J. (1950). Problem-solving processes of college students. Chicago: University of Chicago Press.
Borgman, C. L. (1984). Psychological research in human-computer interaction. In M.E. Williams (Ed.), Annual Review of Information Science and Technology (ARIST) 19, 33-64.
Borgman, C. L. (1986). The user's mental model of an information retrieval system: An experiment on a prototype online catalog. International Journal of Man-Machine Studies, 24, 47-64.
Borgman, C. L., Gallagher, A. L., Krieger, D. & Bower, J. (1990). Children's use of an interactive catalog of science materials. In D. Henderson (Ed.), Proceedings of the 53rd ASIS Annual Meeting 27, 55-68.
Borgman, C. L., Walter, V. A. & Rosenberg, J. (1991). The science library catalog project: Comparison of children's searching behavior in hypertext and a keyword search system. In Jose-Marie Griffiths (Ed.), Proceedings of the 54th ASIS Annual Meeting 28, 162-169.
Bradford, D. (1984). Persona in microcomputer documentation. IEEE Transactions on Professional Communication PC27(2), 65-68.
Brockmann, R. J. (1990). Writing better computer user documentation: From paper to hypertext version 2.0. New York: John Wiley.
Bruner, J. S. (1986). Actual minds, possible worlds. Cambridge, MA: Harvard University Press.
Bruner, J. S. (1990). Acts of meaning. Cambridge, MA.: Harvard University Press.
Bugelski, B. R. (1962). Presentation, total time, and mediation in paired associate learning. Journal of Experimental Psychology, 63, 409-412.
Carney, T. F. (1979). Content analysis: A technique for systematic inference from communications. Winnipeg: University of Manitoba Press.
Carroll, J. M. (1990). The Nurnberg funnel: Designing minimalist instruction for practical computer skill. Cambridge, MA: MIT Press.
Carter, R. (1980). Discontinuity and communication. Paper presented at the East-West Center Conference on Communication Theory East and West, Honolulu, Hawaii. (Available from School of Communication, DS-40, University of Washington, Seattle, Washington, 98195.)
Charles, S. K. & Clark, K. E. (1990). Enhancing CD-ROM searches with online updates: An examination of end-user needs, strategies, and problems. College & Research Libraries, 51, 321-28.
Chavarria, L. S. (1982). Improving the friendliness of technical manuals. In Proceedings of the 29th International Technical Communication Conference (pp. w.26-w.28). Washington, D.C.: Society for Technical Communication.
Chen, H. & Dhar, V. (1990). A knowledge-based approach to the design of document-based retrieval systems. In F. H. Lochovsky & R. B. Allen (Eds.), Proceedings of the Conference on Office Information Systems (pp. 281-290). New York: ACM.
Clever, E. C. & Dillard, D. P. (1991). What do CD-ROM users really need? Information Services & Use, 11, 141-153.
Cooper, E. H., and Pantle, A. J. (1967). The total-time hypothesis in verbal learning. Psychological Bulletin, 68, 221-234.
Coulthard, M. (1977). An introduction to discourse analysis. London: Longman.
Cronbach, L. J. & Meehl, P. E. Construct validity in psychological tests. Psychological Bulletin, 52, 282-302.
Dalrymple, P. W. (1990). Retrieval by reformulation in two library catalogs: Toward a cognitive model of searching behavior. Journal of the American Society for Information Science, 41, 272-281.
Dervin, B. (1983). Information as a user construct: The relevance of perceived information needs to synthesis and interpretation. In S. A. Ward & L. J. Reed (Eds.), Knowledge structure and use: Implications for synthesis and interpretation. Philadelphia: Temple University Press.
Dervin, B. (1991). Information as non-sense; information as sense: The communication technology connection. Paper prepared for book resulting from Sommatie '91 Conference, Veldhoven, The Netherlands. March 19-20. Mimeographed document, Department of Communication, Ohio State University, Columbus, Ohio.
Dervin, B. & Nilan, M. (1986). Information needs and uses. In M.E. Williams (Ed.), Annual Review of Information Science and Technology (ARIST), 21, 3-33.
Diener, C. I. & Dweck, C. S. (1978). An analysis of learned helplessness: Continuous changes in performance, strategy, and achievement cognitions following failure. Journal of Personality and Social Psychology, 36, 451-462.
Doheny-Farina, S. (Ed.). (1988). Effective documentation: What we have learned from research. Cambridge, MA: The MIT Press. Dusenbury, C. et al., (Eds.). (1991). Read this first: An owner's guide to the new model statement of objectives for academic bibliographic instruction. Chicago: ACRL/ALA.
Ensor, P. (1992). Knowledge level of users and nonusers of keyword/Boolean searching on an online public access catalog. RQ, 31(1), 60-74.
Fenichel, C. H. (1981). Online searching: Measures that discriminate among users with different types of experiences. Journal of the American Society for Information Science, 32, 23-32.
Fidel, R. (1991a). Searchers' selection of search keys: I. The selection routine. Journal of the American Society for Information Science, 42(7), 490-500.
Fidel, R. (1991b). Searchers' selection of search keys: II. Controlled vocabulary or free-text searching. Journal of the American Society for Information Science, 42(7), 501-514.
Fidel, R. (1991c). Searchers' selection of search keys: III. Searching styles. Journal of the American Society for Information Science, 42(7), 515-527.
Fleming, H. R. A. (1985). Information behavior among first-year undergraduate students in basic English and media studies courses at the Dorset Institute of Higher Education. Unpublished master's thesis, City University, London.
Frisse, M. E. & Cousins, S. B. (1989). Information retrieval from hypertext: Update on the dynamic medical handbook project. Hypertext '89 Proceedings. (pp. 199-212). New York: ACM.
Gardner, H. (1985). The mind's new science: A history of the cognitive revolution. New York: Basic Books.
Gergen, K. J. (1982). Toward transformation in social knowledge. New York: Springer-Verlag.
Goodman, N. (1984). Of mind and other matters. Cambridge, MA: Harvard University Press.
Habermas, J. (1984). The theory of communicative action Volume 1 Reason and the rationalization of society. Boston: Beacon Press.
Harter, S. P. (1992). Psychological relevance and information science. Journal of the American Society for Information Science, 43(9), 602-615.
Hergenhahn, B. R. (1990). An introduction to theories of personality. Englewood Cliffs, NJ: Prentice-Hall.
Herot, C. F. (1980). Spatial management of data. ACM Transactions on Database Systems, 5, 493-514.
Hildreth, C. R. (1989) Appropriate user interfaces for subject searching in bibliographic retrieval systems. Bookmark, 48(3), 186-193.
Hollnagel, E. & Woods, D. D. (1983). Cognitive systems engineering: New wine in new bottles. International Journal of Man-Machine Studies, 18, 583-600.
Ingwersen, P. (1982). Search procedures in the library analyzed from the cognitive point of view. Journal of Documentation, 38, 165-191.
Jacobson, T. L. (1991). Sense-making in a database environment. Information Processing & Management, 27, 647-657.
Jakobovits, L. A. & Nahl-Jakobovits, D. (1987). Learning the library: Taxonomy of skills and errors. College & Research Libraries, 48(3), 203-214.
James, R. (1983). Libraries in the mind: How can we see users' perceptions of libraries. Journal of Librarianship, 15, 19-28.
Kamala, T. N. (1991). Individual differences in the use of CD ROM databases. Unpublished doctoral dissertation, University of Hawaii, Honolulu.
Katzin, E. (1985). How to write a really good user's manual. New York: Van Nostrand Reinhold.
Keen, M. E. (1977). On the processing of printed subject-index entries during searching. Journal of Documentation, 33(4), 266-276.
Kelman, H. C. (1958). Compliance, identification and internalization: Three processes of attitude change. Journal of Conflict Resolution, 2, 51-60.
Kelly, G. A. (1963). A theory of personality: The psychology of personal constructs. New York: Norton.
Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (Eds.). (1964). Taxonomy of educational objectives: The classification of educational goals. Handbook II: Affective domain. New York: David McKay.
Kuhlthau, C. C. (1988). Developing a model of the library search process: cognitive and affective aspects. RQ, 28(2), 232-242.
Kuhlthau, C. C. (1991). Inside the search process: information seeking from the user's perspective. Journal of the American Society for Information Science, 42(5), 361-371.
Kuhlthau, C. C., Turrock, B. J., George, M. W. & Belvin, R. J. (1990). Validating a model of the search process: A comparison of academic, public and school library users. LISR, 12(5), 5-31.
Labov, W. & Fanshel, D. (1977). Therapeutic discourse: Psychotherapy as conversation. New York: Academic Press.
Larson, R. R. (1983). Users look at online catalogs. Part 2: Interacting with online catalogs, Final report. Washington, D.C.: Council on Library Resources. (ERIC Document ED 231-401)
Lederman, L. (1979). Communication apprehension and the acquisition of information in academic libraries. Unpublished doctoral dissertation, Rutgers University.
Lee, C. (1984 ). Efficacy expectations and outcome expectations as predictors of performance in a snake-handling task. Cognitive Therapy and Research, 8(5), 509-516.
Locke, E. A., Frederick, E., Lee, C. & Bobko, P. (1984). Effect of self-efficacy, goals, and task strategies on task performance. Journal of Applied Psychology, 69(2), 241-251.
LOEX, Library Orientation-Instruction Exchange Clearinghouse. (1992). Ypsilanti, MI: Eastern Michigan University.
Mangrum, R. & Bucknall, T. (1992). U-search: A user study of the CD-ROM service at the University of North Carolina at Chapel Hill. RQ, 31(4), 542-553.
Marchionini, G. (1992). Interfaces for end-user information seeking. Journal of the American Society for Information Science, 43, 156-163.
Markey, K., (1984). Offline and online user assistance for online catalog searchers. Online, 8(3), 54-66.
Mares, G. C. (1909). The history of the typewriter, successor to the pen: An illustrated account of the origin, rise, and development of the writing machine. London: Gilbert Putnam. (Reprinted Arcadia, CA: Post-era Books, 1985.)
Margolis, N. (1989). Making the manual work: Techniques for validating computer documentation. Performance and Instruction, 28(6), 1-9.
Martin, T. H. & Wyman, J. C. (1983). Feedback and exploratory mechanisms for assisting library staff improve on-line catalog searching: Final report. Washington, D.C.: Council on Library Resources. (ERIC Document, ED 258-577)
Meadow, C. T. (1988). OAKDEC: A program for studying the effects on users of a procedural expert system for database searching. Information Processing and Management, 24, 449-457.
Mead, G. H. (1934). In C. Morris, (Ed.). Mind, self, and society. Chicago: University of Chicago Press.
Meichenbaum, D. H. & Smart, I. (1971). Use of direct expectancy to modify academic performance and attitudes of college students. Journal of Counseling Psychology, 18, 531-535.
Mellon, C. (1986). Library anxiety: A grounded theory and its development. College & Research Libraries, 47, 160-165.
Merrill, M. D. (1972). Psychomotor taxonomies, classifications, and instructional theory. In R N. Singer, (Ed.), Psychomotor domain: Movement behaviors (pp. 385-414). Philadelphia: Lea and Febiger.
Mirel, B. (1988). The politics of usability: The organizational functions of an in-house manual. In S. Doheny-Farina, (Ed.). Effective documentation: What we have learned from research (pp. 277-298). Cambridge, MA: The MIT Press.
Mischel, W. (1993). Introduction to personality. Philadelphia: Harcourt Brace Jovanovich.
Montgomery, D. C. (1991). Design and analysis of experiments. New York: John Wiley.
Mozeico, H. (1982). A human/computer interface to accomodate user learning stages. In J. A. Larsen (Ed.), Tutorial: End user facilities in the 1980's (pp. 442-446). New York: IEEE.
Nahl-Jakobovits, D. & Jakobovits, L. A. (1992). Content analysis method for developing user-based objectives. Research Strategies, 10, 4-16.
Nahl-Jakobovits, D. & Jakobovits, L. A. (1993). Bibliographic instructional design for information literacy: Integrating cognitive and affective objectives. Research Strategies, 10(4) (in press).
Naismith, R. & Stein, J. (1989). Library jargon: Student comprehension of technical language used by librarians. College & Research Libraries, 50(5), 543-552.
Neale, J. M. & Liebert, R. M. (1986). Science and behavior: An introduction to methods of research. Englewood Cliffs, NJ: Prentice-Hall.
Norman, D. A. (1988). The psychology of everyday things. New York: Basic Books. Norman, D. A. & Draper, S. W. (1986). User centered system design: New perspectives on human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum.
Page, M. (1993). A personal view of the Internet. College & Research Libraries News, 54(3), 127-132
Pool. I. deS., (Ed.). (1959). Trends in content analysis. Urbana, IL: University of Illinois Press.
Powell, G.E. (1973). Negative and positive mental practice in motor skill acquisition. Perceptual and Motor Skills, 37, 312-313.
Reder, L. M. (1982). Elaborations: When do they help and when do they hurt? Text, 2, 211-224.
Reder, L. M., Charney, D. H., & Morgan, K. I. (1986). The role of elaborations in learning a skill from an instructional text. Memory & Cognition, 14, 64-78.
Reese, J. & Steffey, R. (1987). ERIC on CD-ROM: A comparison of DIALOG OnDisc, OCLC's Search CD450 and SilverPlatter. Online, 11(5), 42-54.
Roberts, R. N. (1979). Private speech in academic problem-solving: A naturalistic perspective. In G. Zevin (Ed.), The development of self-regulation through private speech (pp.295-323). New York: Wiley.
Roberts, R. N. & Tharp, R. G. (1980). A naturalistic study of children's self-directed speech in academic problem-solving. Cognitive Research and Therapy, 4, 341-353.
Rogers, C. R. (1966). Client-centered therapy. In S. Arieti (Ed.), American Handbook of Psychiatry: Vol. 3 (pp. 183-200). New York: Basic Books.
Rogers, C. R. (1969). Freedom to learn. Columbus, OH: Charles E. Merrill.
Rogers, C. R. (1951). Client-centered therapy: Its current practice, implications and theory. Boston: Houghton Mifflin.
Sandore, B. (1990). Online searching: What measure satisfaction? LISR, 12, 33-54.
Saracevic, T. & Kantor, P. (1988). A study of information seeking and retrieving: III. Searchers, searches, and overlap. Journal of the American Society for Information Science, 39, 197-216.
Seligman, M. E. P. (1990). Learned optimism. New York: A. A. Knopf.
Searle, J. R. (1969). Speech acts: An essay in the philosophy of language. London: Cambridge University Press.
Searle, J. R. (1983). Intentionality: An essay in the philosophy of mind. New York: Cambridge University Press.
Shaw, D. (1993). CD-ROM interfaces for information retrieval: Heuristic evaluation and observations of intended users. Proceedings of the 14th National Online Meeting, 14, 371-377.
Shneiderman, B. (1980). Software psychology: Human factors in computer and information systems. Cambridge, MA: Winthrop.
Shneiderman, B. (1987). Designing the user interface: Strategies for effective human-computer interaction. Reading, MA: Addison-Wesley.
Staats, A. W. (1975). Social behaviorism. Homewood, IL: Dorsey.
Stewart. L. Helping students during online searches: An evaluation. Journal of Academic Librarianship, 18(6), 347-351.
Swedenborg, E. (1975). Arcana celestia (Vol. 4) (J. F. Potts, Trans.). New York: Swedenborg Foundation. (Original work published 1751)
Swedenborg, E. (1977). Spiritual diary (Vol. 4) (G. Bush and J. F. Buss, Trans.). New York: Swedenborg Foundation. (Original work published 1843)
Taylor, R. S. (1986). Value-added processes in information systems. Norwood, NJ: Ablex.
Tenopir, C. & Lundeen, G. (1988). Managing your information: How to design and create a textual database on your microcomputer. New York: Neal-Schuman.
Tenopir, C., Nahl-Jakobovits, D. & Howard, D. L. (1990). Full-text search strategies and modifications: The role of the searcher and the role of the system. Proceedings of the Eleventh National Online Meeting 11, 389-399.
Tenopir, C., Nahl-Jakobovits, D. & Howard, D. L. (1991). Strategies and assessments online: Novices' experience. Library and Information Science Research, 13(3), 237-266.
Tenopir, C. & Neufang. R. (1992a). Electronic reference options: How they stack up in reference libraries. Online, 16(1), 22-28.
Tenopir, C. & Neufang, R. (1992b). The impact of electronic reference on reference librarians. Online, 16(3), 54-60.
Tumlin, M. D. (1993). Point-of use instruction on CD-ROM: How much is too much? Proceedings of the 14th National Online Meeting 14, 415-421.
Turner, J. H. (1992). The structure of sociological theory, 5th ed. Belmont, CA: Wadsworth.
Tversky, A. & Kahneman, D. (1983). Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90, 293-315.
Vendler, Z. (1967). Linguistics in philosophy. Ithaca, NY: Cornell University Press.
Vygotsky, L. S. (1965). Thought and language (E. Hantmann & G. Vokar, Eds. & Trans.). Cambridge, MA: MIT Press.
Whitaker, C. S. (1990). Pile-up at the reference desk: Teaching users to use CD-ROM. Laserdisk Professional, 3(2), 30-34.
Winograd, T. (1977). A framework for understanding discourse. In M. A. Just & P. A. Carpenter (Eds.). Cognitive processes in comprehension (pp. 63-88). Hillsdale: Earlbaum. Woelfle, N. N. (1984). Individual differences in online search behavior: The effect of learning styles and cognitive abilities on process and outcome. Unpublished doctoral dissertation, Case Western Reserve University, Cleveland, OH.
Wurman, R. S. (1990). Information anxiety: What to do when information doesn't tell you what you need to know. New York: Doubleday Bantam Books.
Zloof, M. M. (1977). Query-by-example: A database language. IBM Systems Journal, 16, 324-343.