Elisabeth
Kittredge
Anne
Hillier
LIS
665
Spring
2005
Dr.
Diane Nahl
LILO Rubrics Project: LILO Module 2
Introduction
Rubrics are a useful tool for authentic assessment. Although they have traditionally been used at
the K-12 level, they are emerging as an important tool for assessing
information literacy at the collegiate level as well. Rubrics are useful because they encourage
objective, rather than subjective, assessment.
Rubrics specify criteria by which students’ work will be assessed, and
for each item, enumerate performance standards across a range of levels of
competency, thereby enabling instructors to measure students against a set
standard rather than relative to other students’ performance. When students are informed of the evaluation
criteria, it can help to increase student motivation, as students can expect
their work to be judged equitably.
Rubrics can also help increase student confidence because students are
aware of what the instructor expects of them (Jacobson and Xu, 108). However, in the LILO Rubric Project, students
are not aware of the rubric standards, so these benefits are not maximized. Although
we created the LILO rubrics for ease of use for the instructors rather than for
the students, we believe that the rubrics could also be used by the students
and would be of value to them.
Procedures
In this project, we created analytic rubrics (or what
California State University at Fresno refers to as primary trait analysis) for the prompts of LILO Module 2. We were given Megan Oakleaf’s dissertation
rubric as a model, and followed the steps for developing rubrics suggested by
Bonnie Gratch-Lindauer (32-33). In order
to maintain uniformity among the rubrics created by different groups, we used
the scores and levels of competency set by the instructor, Beginning (0),
Beginning (1), Proficient (2), and Advanced (3).
First, we mapped the prompts of LILO Module 2 to the ACRL
Information Literacy Standards (see Table 1). LILO prompts 2.1-2.6 were already
mapped on pages 17-18 of the assignment instructions, but we added performance indicators for LILO prompts
2.7-2.10. Next, we identified the tasks
students were being asked to perform by each prompt and used them to write the
characteristics to be evaluated by each rubric.
In some cases, we clustered related prompts into a single rubric.
Examining student responses, we defined standards for
each level of each rubric. We
extracted some of the best and worst responses and added them to the rubrics as
examples of Beginning and Advanced work, then found examples of intermediate
quality and added them as examples of Proficient work. This step was in accordance with
Jacobson and Xu, who suggest using examples to create rubrics (109).
Table 1: LILO Prompts and Corresponding ACRL
Performance Indicators
|
LILO
Prompt |
ACRL
Performance Indicators* |
|
LILO 2.1 What
important verbs are included in the description of your assignment? |
PI 1.1. The information
literate student defines and articulates the need for information. PI 1.2. The information
literate student identifies a variety of types and formats of potential
sources for information. |
|
LILO
2.2 What kind of approach do those
verbs indicate? |
Same as above. |
|
LILO 2.3 Is
there more than one part to your assignment? What are the main parts? |
Same as above. |
|
LILO
2.4 Briefly describe any special instructions given for this
assignment. |
Same as above. |
|
LILO
2.5 Besides your instructor, who is the audience for your
assignment? |
PI 1.4. The information
literate student reevaluates the nature and extent of the information need. |
|
LILO 2.6 Can you assume your audience already knows about your
topic? What specific opinions or biases might they have? |
Same as above. |
|
LILO
2.7 Are books useful resources for your topic? Why or why
not? |
PI
1.2.c. The information literate student identifies the value and differences
of potential resources in a variety of formats. |
|
LILO 2.8 Are
scholarly journal articles useful resources for your topic? Why or why
not? |
Same as above. |
|
LILO 2.9 Are
popular magazine articles useful resources for your topic? Why or why
not? |
Same as above. |
|
LILO 2.10 Are
Web sites useful resources for your topic? Why or why not? |
Same as above. |
*All
Performance Indicators used here fall under ACRL Information Literacy Standard
1: The information literate student determines the nature and extent of the
information needed.
Discussion
Both Relearning by Design and California State University
at Fresno advise that descriptions of the best performance should be written
first, but we often found our worst examples first. However, contrasting the best and worst
student work made it easier to define standards, and in all rubrics, we
followed the pattern of writing descriptions of the extremes first and the
intermediate level last.
Our rubrics are good in that they are very thorough,
clearly defining criteria and standards in “parallel language” (suggested by
Relearning by Design) as well as giving examples for each level, unlike the
“Acquiring Information Rubric” (Holt, Rinehart, and Winston), which does not give
examples. Relearning by Design also suggests that
rubric creators “use 2-3 different samples of excellence so as not to limit
your or your students’ thinking about possible excellence,” and we give
multiple examples for some levels to show different approaches that are equally
acceptable. However, we had some
difficulty defining standards and selecting examples for the Proficient level,
as “proficient” responses were often those that were somewhat acceptable but
not outstandingly good.
Referring to student work when creating our rubrics
enabled us to tailor the rubrics to LILO’s users, college freshmen in required
writing courses. Prompts 2.3 and 2.4
were worded ambiguously with some overlap between the two, and students
interpreted the two in different ways, so we combined their assessment into one
rubric. It might also be logical to
assess the related prompts 2.1 and 2.2 with a single rubric. However, with two rubrics the assessment data
is clearer. It is easier to see that
many students do not know what a verb is.
Based on student responses, we decided that students
should be rewarded for effort, and as a class, we decided on a scoring system
that had both zero- and one-point scores for the Beginning level. However, since the Proficient and Advanced
levels are only associated with a single score, the rubrics might be more
logical and easier to use if a fourth level—for example, Incomplete--were
created for the zero score.
According to Relearning by Design’s mandate that “rubrics
must be continuous,” the difference between Beginning and Proficient
performance should be equal to the difference between Proficient and Advanced
performance. While our examples and
descriptions show continuous improvement across the levels, the names assigned
to the levels do not. This sometimes
causes a discrepancy between performance characteristics and the name of a level, as in Table 2: Rubric for Identifying Important Verbs. A student who, when asked to extract
verbs, extracts “some verbs [and] some non-verbs” is performing at what we
determined to be an intermediate level, but an instructor would hardly call
this imperfect response “proficient.”
“Proficient” implies that the student consistently performs at a
sufficient level of competence. A
beginner’s growth to a “proficient” level of competence often requires a great
increase in skill, greater than that between “proficient” and the rather vague
term “advanced.” In contrast, Megan
Oakleaf’s LOBO rubric uses the levels Beginning, Developing, and
Exemplary—“developing” is clearly an intermediate stage with further room for
improvement, while “exemplary” work is clearly outstanding. The Colorado State Department of Education
Rubrics for the Assessment of Information Literacy use the levels In Progress,
Essential, Proficient, and Advanced, setting an intermediate level between the
lowest level and “proficient.”
Another criticism that might be made is that since these rubrics were
designed for primary trait analysis, scoring
is not balanced across the module. The
related tasks of identifying verbs and deducing the approach they indicate are
split into two rubrics and together receive a maximum score of
six points, while the four prompts about the usefulness of books, journal
articles, popular magazine articles and websites are evaluated with a single
rubric, so four responses only receive a maximum score of three points. However, these rubrics can be adapted to produce a more
balanced holistic assessment by weighting scores or by
modifying the rubrics (for example, duplicating the Rubric for Identifying Useful Resources into
four similar rubrics, one for each prompt).
As stated earlier, these rubrics
could be of greater use if they were shown to students instead of only being
used by instructors. When students
understand how they will be evaluated and what is expected of them, it
increases their confidence and motivation (Jacobson and Xu, 108). However, if students have copies of the
rubrics as they go through LILO, they may give less independent thought to the
prompts and merely parrot the examples given in the rubrics.
Rubrics
Table 2: Rubric for Identifying Important Verbs
|
CRITERIA |
BEGINNING |
PROFICIENT |
ADVANCED |
ACRL |
|
Identifies important verbs |
0: No answer. 1: Student extracts only
non-verbs. |
2: Student extracts some
verbs, some non-verbs. |
3: Student extracts only
verbs. |
LILO 2.1 Identifies
key concepts and terms that describe the information need (ACRL 1.1.e) |
|
Examples |
0: No answer. 1: a) fast food b) prevention |
2: a) informative, explain b) cruelty, testing,
analyze, alternatives |
3: a) describe, compare b) gather, evaluate,
draft, revise, edit, find, formulate, read, and organize. |
Table 3: Rubric for Articulating Approach Indicated
by Verbs
|
CRITERIA |
BEGINNING |
PROFICIENT |
ADVANCED |
STUDENT LEARNING OUTCOMES |
|
Articulates approach indicated by verbs |
0: No answer. 1: Student does not show
comprehension of approach indicated by verbs. |
2: Student shows some comprehension of approach indicated by
verbs. |
3: Student shows full comprehension of approach
indicated by verbs. |
LILO 2.2 The information literate
student defines and articulates the need for information. (ACRL 1.1) The information literate
student identifies a variety of types and formats of potential sources for
information. (ACRL 1.2) |
|
Examples |
0: No answer. 1: a) [Verbs: describe,
compare, argue, analyze] Approach:
contribute and prevent stroke b) [Verbs: compare,
relate] Approach: medical |
2: [Verb: analyze] Approach: really take a close look at the
topic |
3: [Verbs: argue, analyze,
compare, describe] Approach: argue, requires that you take a side, analysis
requires a close examination of something, compare and contrast, requires you
to look for similarities and differences between two or more things, and draw
conclusion |
Table 4: Rubric for Identifying Parts of Assignment
and Special Instructions
|
CRITERIA |
BEGINNING |
PROFICIENT |
ADVANCED |
STUDENT LEARNING OUTCOMES |
|
Identifies parts of assignment and special
instructions. |
0: No answer.
|
2: Student shows some awareness
of parts of assignment or special instructions but not both. |
3: Student shows full awareness of parts of assignment and
special instructions. |
LILO 2.3, 2.4 The information literate
student defines and articulates the need for information. (ACRL 1.1) The information literate
student identifies a variety of types and formats of potential sources for
information. (ACRL 1.2) |
|
Examples |
0: No answer. 1: Parts: interview |
2: Parts: Write a short
essay about syphillis and Gohonereah and how it impacted the hawaiians when
cook arrived. Special
Instructions: Use time lines and geographical areas |
3: a) Parts: Establish issue, provide support for both
sides and then state your own stance. Special Instructions: No more than six pages, APA guidelines. b) Parts: Yes there is more
than one part to this assignment, research, draft then final draft. Special Instructions: Do MLA
format. c) Parts: paper,
presentation using visual presentational aids Special Instructions: 8-10 pages, APA style, minimum
of 8 professional reference articles |
Table 5: Rubric for Identifying Audience
|
CRITERIA |
BEGINNING |
PROFICIENT |
ADVANCED |
STUDENT LEARNING OUTCOMES |
|
Identifies audience |
0: No answer. 1: Student does not consider possibility of greater audience
beyond classmates |
2: Student shows awareness of audience beyond classmates but
does not articulate who that audience may be |
3: Student shows awareness of impact of content on a greater
audience |
LILO 2.5 The information literate
student reevaluates the nature and extent of the information need. (ACRL 1.4) |
|
Examples |
0: No answer, not even
“classmates” 1: classmates |
2: a) college level students b) Members of the general
public. |
3: a) Mothers and Fathers
of Bottlefed Babies b) Hawaiian students in
school and out of school c) Para-professionals whose
interest are in the field of education. |
Table 6: Rubric for Evaluating Audience’s Knowledge
and Biases
|
CRITERIA |
BEGINNING |
PROFICIENT |
ADVANCED |
STUDENT LEARNING OUTCOMES |
|
Evaluates audience’s knowledge and biases |
0: No answer. 1: Student is unaware of audience’s knowledge and biases. |
2: Student has a general idea of audience’s knowledge and biases |
3: Student clearly states audience’s knowledge and biases. |
LILO 2.6 The information literate
student reevaluates the nature and extent of the information need. (ACRL 1.4) |
|
Examples |
0: No answer 1: yes, not sure. |
2:
a) Just what they hear on the
news/media. b) They know very little
about the topic. c) no, they will have their own self experiences |
3: a) I
can not assume that my audience knows anything about my topic. The purpose is
to inform the audience and assuming that they do not know anything about my
topic. b)
Possibly. most of my audience might have the opinion that hazing occurs only
in Fraternety's, or that only certain people will get involved with it. c) no,
They may already be bias against the space program due to rising costs to
fund it |
Table 7: Rubric for Identifying Useful Resources
|
CRITERIA |
BEGINNING |
PROFICIENT |
ADVANCED |
STUDENT LEARNING OUTCOMES |
|
Identifies useful resources for topic |
0: No answer. 1: Student is vague about why resources would be useful to
them, does not show understanding of strengths and limitations. |
2: Student shows some understanding of strengths and
limitations of some resources. |
3: Student shows thorough understanding of strengths and
limitations of all resources; evaluates
usefulness for topic. |
LILO 2.7-2.10
|
|
Examples |
0: No answer. 1: a) [of books] yes. if i can find a book, then at least i know
the information isnt made up. b) [of websites] Yes, easily attained info 1: [of journals] yes
because they have good information |
2: [of websites] only if
it is a safe website, like that of the CDC. |
3: [of books] not really, PGD is a very new and high
tech; journals would be a better option [of journals] Yes. The information is current and
credible. In addition the assignment is to use peer reviewed articles [of magazines] No. The information in these
magazines is not reliable. For this paper, more scientific resources are
needed. [of websites] Yes, however, it is important to be
sure the web site is credible because there is a lot of junk out there. Web
sites like .gov and .edu are credible. |
Future Research
We
discovered when reviewing the student responses that there was confusion as to
the meaning of two of the prompts. For
LILO prompt 2.1, which asked the students to list the important verbs included
in the description of the assignment, several students wrote down answers that
were composed of adjectives and nouns, but not verbs. In order to clear up any confusion as to
parts of speech, we suggest that the LILO prompt 2.1 should read "What
important verbs ("action words") are included in the description of
your assignment?" The addition of
this clarifier would help a student who has difficulty with parts of speech
complete this portion of the assignment.
We
also found LILO 2.3 to be ambiguous.
LILO 2.3 asks "Is there more than one part to your assignment? What are the main parts?" We discovered from looking at student answers
that there were several interpretations to this set of questions. Many students responded by giving the
physical parts of the assignment. For
example, several students stated that the parts of the assignment were the
introduction, body, conclusion, and references.
Other students thought of the process that they would need to go
through, rather than the physical end product.
They referred to such things as the outline, bibliography, rough draft,
and final draft. In order to make the
prompt more comprehensible examples of parts of the assignment should be given
or the prompt should be rewritten. The
prompt could be rewritten as "What are the major sections of the
paper/project? (i.e. introduction, body, conclusion, references)" or
"What are the steps that you will need to complete? (i.e. outline,
bibliography, rough draft, final draft)."
The choice of revised prompt will depend upon what the LILO developers
want the students to share about their research assignment.
Librarians are being asked to do
more and more instruction as well as create more online instruction tools. With the shift to standards-based
instruction, it is imperative that librarians are able to assess the impact of
their teaching and online-tutorials on student learning. As we discovered during lecture on the tenure
process, it is very important that librarians are able to justify the impact
their instruction is having on student learning. It is also important that the entire library
is able to show its value when budget decisions, and especially budget cuts,
are made. Just as importantly, rubrics
provide tangible results to show how outcomes are being met to accrediting bodies.
Lorrie A. Knight, a librarian at the
University of the Pacific at Stockton in California, believes that rubrics are
a useful assessment tool. She used
rubrics to assess a bibliography project and found that rubrics "can
produce a body of data that can be easily understood and reported to interested
stakeholders" (Knight, 52). She
also reports an additional value in using rubrics. The online tutorial was revised based on the
results of the study from the rubrics (Knight, 52). Knight also discusses the changing relationship between librarians and
patrons. Technology is creating an
"educational distance" in which librarians are no longer able to
visibly witness if a student has gained mastery of a skill (Knight, 53). Knight stresses that is
vital that empirical evidence is gathered that can determine if a skill has
been mastered (53). Rubrics are an
authentic means of assessing mastery.
The use of rubrics began at the K-12
level, but their use is gradually trickling up into higher education. Rosemarie Bernier, a library media teacher at
Alexander Hamilton High School in the Los Angeles Unified School District,
suggests that librarians can make themselves indispensable by helping teachers
create rubrics (24). Although her
article was written for the K-12 audience, her theory is applicable to higher
education. She states that librarians should help teachers make rubrics
because this collaborative exercise helps the teachers create standards-based
instruction and "provides us (the librarians) with tools for documenting
how we improve instruction" (Bernier, 25).
Partnerships between librarians and faculty could be developed at the
collegiate level which would increase the visibility and status of librarians.
As future academic librarians, this
assignment of creating rubrics from student responses to the LILO online
tutorial has been of incredible value.
We are now familiar with creating an assessment tool to gauge student
learning that we can use to show our impact on student learning. Having created one set of rubrics for this
class, it will be easier for us to create others in the future. We now understand the underlying concepts of
what a rubric is, the importance of having rubrics, how to extract criteria and
examples of criteria, as well as how to define degrees of competency. In addition, like all of the assignments we
have created for this class we are able to tie our rubrics to the ACRL
standards.
References
American Library
Association. “Information Literacy
Competency Standards for Higher Education” Chicago, IL: ALA/ACRL, 2000.
Bernier, Rosemarie. “Making Yourself
Indispensable by Helping Teachers Create Rubrics,” CSLA Journal 27
(2): 24-25.
California State University
at Fresno. “Teaching, Learning and Technology: Using Scoring Rubrics.” 2002. Internet: available from http://www.csufresno.edu/cetl/assessment/UsingScoringRubrics.pdf Accessed May 1, 2006.
Colorado State Department of
Education. “Rubrics for the Assessment
of Information Literacy.” Colorado
State Dept. of Education, Denver. State Library and Adult Education Office. 1996.
Internet: available from http://www.eric.ed.gov/ERICDocs/data/ericdocs2/content_storage_01/0000000b/80/23/bf/d9.pdf Accessed May 1, 2006.
Holt, Rinehart, and
Winston. “Rubric 1: Acquiring
Information.” Internet: available from
http://www.sites4teachers.com/links/redirect.php?url=http://go.hrw.com/resources/go_ss/teacher99/rubrics/RUBRIC01.pdf Accessed May 1, 2006.
Gratch-Lindauer,
Bonnie. “Selecting and Developing
Assessment Tools.” Chapter 3 in Elizabeth Fusler Avery, Ed., Assessing Student Learning Outcomes for
Information Literacy Instruction in Academic institutions. Chicago, IL:
ALA/ACRL, 2003, pp. 22-39.
Jacobson, Trudi E. and
Lijuan Xu. Motivating Students in Information Literacy Classes. New York: Neal-Schuman Publishers, Inc.,
2004.
Knight, Lorrie A. “Using Rubrics to Assess Information
Literacy,” Reference Services Review 34
(1): 43-55.
Oakleaf, Megan. LOBO Dissertation Rubric. Created March 23, 2006. Provided
to LIS 665 by Dr. Diane Nahl, April
2006.
Relearning by Design,
Inc. “What is a Rubric?” Internet: available from
http://www.relearning.org/resources/PDF/rubric_sampler.pdf Accessed May 1, 2006.