Margot Hanson, Megan Radomski, Jacob Rosen, Xiangli Xu
1 May 2006, LIS 665, Nahl
LILO Rubrics Project
In our project we hoped to create a series of rubrics that would evaluate students’ performance while using module 5, Evaluating Resources, within the University of Hawaii’s LILO (Learning Information Literacy Online), an interactive online tutorial that teaches students basic concept of research. But in order to understand the Rubrics we created and the process we used, it is important to consider the literature that informed our work.
The Center for Enhancement of Teaching and Learning (CETL) in California State University, Fresno, gives their definition of scoring rubrics as scoring rubrics are explicit schemes for classifying products or behaviors into categories that vary along a continuum[1]. Using scoring rubrics, the instructor can examine student’s project efficiently. In CETL’s scoring rubric file, they suggested how to make scoring rubrics, they suggested to top category and lowest acceptable category first, then develop the intermediate level category.
Using rubrics for evaluation of student performance has
two distinct advantages: rubrics describe and precisely outline levels of
achievement[2]. One disadvantage is that their construction can be
time consuming. Descriptors in rubrics must be precise, while maintaining the
balance between vagueness and complexity. In order to streamline this process
we followed several guidelines that mimic those set forth by San Diego State
University (SDSU). The SDSU guide instructs rubric creators to first determine
learning outcomes, keep the rubric short and simple, and focus on measurable
criteria[3].
Rubrics are only recently being
integrated into higher education instruction. Throughout our design process, we
relied heavily on Megan Oakleaf's rubric table for LOBO (The Library Online
Basic Orientation), the system that LILO is based on[4]. We used her
basic overall layout, the method of grouping our prompts into more general
evaluation criteria, and her organizational structure for the student learning
outcomes section, in which we listed the applicable LILO prompts, followed by
the learning outcomes for the rubric, and finally the relevant ACRL
standard/performance indicator/learning outcome code numbers.
Our LILO module for this project was section 5, which deals with Evaluating Sources. There are four parts to this module, which focus on books, articles, web sites, and selecting useful information. The first three parts each have a few prompts which allow the students to type feedback into the box on the screen, but the last part, on selecting useful information, only has text for the student to read. The prompts we examined and created rubrics for were from the first three parts. Following is a list of the eighteen prompts we analyzed:
5.1 The exercises in this section will help you evaluate a specific book. Type the title and author of the book you will evaluate here:
5.2 Answer the questions above for the book you’re evaluating. Overall, does what you know about the author/authority of the book indicate that it’s a good resource? How can you tell?
5.3 Answer the questions above for the book you’re evaluating. Overall, does what you know about the purpose of the book indicate that it’s a good resource? How can you tell?
5.4 If there is a table of contents, which chapters would be most relevant to your topic focus? Why?
5.5 If there is an index, what keywords will lead you to information about your topic?
5.6 If there is a bibliography, which entries might be relevant to your topic focus?
5.7 Overall, does what you know about the publisher of the book indicate that its a good resource?
5.8 Overall, does what you know about the date of publication indicate that it's a good resource? How can you tell?
5.9 Overall, is this book a good resource to use for your assignment? Why?
5.10 Describe—in detail and in your own words—the difference between scholarly journal articles and popular magazine articles.
5.11 Why do you think instructors require students to use scholarly journal articles instead of popular magazine articles?
5.12 The exercises in this section will help you evaluate a specific Web site. Type the title and URL (Web address) of the Web site you will evaluate here:
5.13 Based on what you know about the authority of this Web site, is it appropriate to use for your assignment? Why or why not?
5.14 Based on what you know about the purpose of this Web site, is it appropriate to use for your assignment? Why or why not?
5.15 Based on what you know about the currency of this Web site, is it appropriate to use for your assignment? Why or why not?
5.16 Based on what you know about the content of this Web site, is it appropriate to use for your assignment? Why or why not?
5.17 Based on what you know about the bias or point of view of this Web site, is it appropriate to use for your assignment? Why or why not?
5.18 Of all the ways to evaluate a Web site based on authority, purpose, currency, content, or bias/point of view—which do you think is most difficult? Why? What lingering questions might you have about the Web site that it was unable to clearly answer?
To analyze the above prompts and create rubrics to assess student responses, we used the ACRL Information Literacy Standards, Performance Indicators and Outcomes[5] as our guide. Following is a list of the ACRL Standards, Performance Indicators and Outcomes we chose which were particularly relevant to our project.
Standard One: The information literate student determines the nature and extent of the information needed.
Performance Indicator Two: The information literate student identifies a variety of types and formats of potential sources for information.
Learning Outcomes:
a. Knows how information is formally and informally produced, organized, and disseminated
b. Recognizes that knowledge can be organized into disciplines that influence the way information is accessed
c. Identifies the value and differences of potential resources in a variety of formats (e.g., multimedia, database, website, data set, audio/visual, book)
d. Identifies the purpose and audience of potential resources (e.g., popular vs. scholarly, current vs. historical)
Standard Three: The information literate student evaluates information and its
sources critically and incorporates selected information into his or her
knowledge base and value system.
Performance Indicator Two: The information literate student
articulates and applies initial criteria for evaluating both the information
and its sources.
Learning Outcomes:
a. Examines and compares information
from various sources in order to evaluate reliability, validity, accuracy,
authority, timeliness, and point of view or bias
c. Recognizes prejudice, deception, or manipulation
d. Recognizes the cultural, physical, or other context within which the
information was created and understands the impact of context on interpreting
the information
Standard Three, Performance
Indicator Four: The information literate student
compares new knowledge with prior knowledge to determine the value added,
contradictions, or other
unique
characteristics of the information.
Learning Outcomes:
a. Determines whether information satisfies the research or other
information need
e. Determines probable accuracy by questioning the source of the data,
the limitations of the information gathering tools or strategies, and the
reasonableness of the conclusions
Before we defined the three degrees of competency of rubrics, we compared the LILO prompts to ACRL IL standards, performance indicators and outcomes, then we matched each of the LILO prompts with ACRL IL standards, performance indicators and outcomes, then grouped the LILO prompts into five groups based on ACRL IL standard to make our five assessment rubrics. These five rubric groupings are: “Identifies Purpose of Resource,” “Recognizes Bias / Point of View of Resource,” “Evaluates Authority of Resource,” “Evaluates Relevance to Information Need,” and “Understands Organization of Resource.” Our rubrics are presented in the appendix.
First we thought about a basic description of what skills
satisfied the learning outcomes in regards to each rubric. We then went through the student
responses of the prompts we mapped to the outcomes to get a feel for the
different levels of competency.
Next we split the skills into three different levels and proceeded to
look for examples of each. As we
did this, we refined our wording for the degrees of competency.
For example, under the heading “Evaluates Relevance to
Information Need” we integrated the ACRL Outcomes 3.2.d and 3.4.a. These
outcomes deal with the student successfully determining if a source satisfies
their information need and the student’s ability to discern the context of the
information. It was difficult to define levels of competency in this area
because the quality and extent of the students’ answers varied wildly. The core
of this section of our rubric was intended to evaluate the student’s ability to
consider relevance. We separated the degrees based loosely on the answers
students provided, seeing that the best answers were able to articulate how a
source was relevant to their topic, and not that it was merely a good source.
The rubrics we
created measure the degrees of competency for evaluating resources. Each rubric relates to a large number of
prompts and in some cases only sections of the prompts are relevant to the
rubric. Therefore it is difficult
to give a clear picture of what gives an exemplary answer for each level of
competence. In some cases, the
level of competency can only be determined by seeing whether or not the student
is able to locate the desired information, thereby proving their understanding
of the resource's organization, relevance, bias, purpose, or authority.
The section “Evaluates Relevance to Information Need” is clearly flawed because students were not able to consistently achieve the “proficient” and “advanced” levels. Most answers fell into the “beginning” and “proficient” categories. While this spread of information should not necessarily dictate a change in the rubric, it does illuminate the fact that students seem to have a difficult time achieving the goals of LILO in its current state.
If we added one more degree of competency, maybe we could have covered the range of examples better. Other suggestions for change are more closely related to the organization of LILO itself, and follow in the next section on future research.
More research is needed into how students interpret the plethora of questions as well as how they choose which questions to answer. Across all areas there was a high level of inconsistency as to what questions students choose to answer.
We would not structure the prompts so that the student is given four or five questions to answer in one prompt (applies to prompts 5.13, 5.16, as well as many others). Also, the different subdivisions within this module—books, journals and magazines, websites—are not structured in a uniform manner. The prompts about journals are general questions about the organization of journals and magazines without relating to the student’s topic, while the prompts about books and websites ask questions about specific sources that the student is using. In terms of the structure of the site, navigating back and forth can be frustrating as using the “back” button on the browser takes the user completely out of the section they were in, which forces them to click through the “Next Page” buttons to return to where they were (numbering of the sections is also a tad confusing). We are not sure if information gets lost while doing this. Also, the answers on the spreadsheet get cut off if the answers are too long. We do not know if this is just a function of how it was printed or if this would be a problem for preserving the information the student enters into the system.
When many students used LILO they either were not using all the resources that were listed or did not have the appropriate resource in hand. For example, in the book section many students were having trouble finding an appropriate book or were using mainly articles. This indicates that students either need more modularity when using LILO or that they need more help when using it. While sections of LILO can be edited many students either were not aware of this or did not take the time to do so. Also more than a small percentage of students’ answers did not answer the main subject prompt or answered it poorly. Because the prevalence of incomplete answers is such a problem a link to further resources of email reference may be worthwhile.
There are several questions where student responses became a lot of “yes, yes, no, not sure, yes…”
Following are specific suggestions for each of the prompts in our module:
5.1 This prompt is fairly simple. Almost all students that were using a book as a resource were able to locate the title and author. Students not using any books generally pointed out how the prompt was not applicable to them, showing that perhaps the prompt's wording will make it irrelevant to a percentage of student responders.
5.2 There are four questions about the author given on top of the question box. Inside the question box, it asks students one more question about student’s conclusion to the book resource based on the author background. Because those question are not listed together, some of the students answer all of those questions, some of the students answer the top four questions only, some answer the last question only, some didn’t answer any question. This could be improved by clearly indicating which questions should be answered, or providing a response box for each question.
5.3 Many students answered why the book is a good resources but did not relate this back to their topic. It seems that the purpose of the question is unclear to students.
5.4 Virtually no student related the chapters listed to how they could help their topic. The most common answer was to merely list chapter numbers that were applicable to their topic. Most of the introspection that occurred in this question was discussing the scope of the chapters, not how they would be useful for the student’s paper.
5.5 For this question students either listed keywords or they didn’t. This question is difficult to assess because there is no way to tell whether the words they chose are the most applicable of all the applicable words.
5.6 Very few students actually copied a useful amount of a citation. Less than half a dozen. Most students gave yes or no answers indicating the existence of a bibliography, not useful entries. It seems that this question is confusing to students. They need the term bibliography more thoroughly explained and possibly a system of boxes (or a citation builder) to type the applicable citations into their journals. The answers they gave would not be helpful if they wanted to find additional resources.
5.7 The explanation about the publisher can help student understand how to answer the question. But the problem is the same as question 5.2, one question is given on top of the question box with the question explanation, and one question inside question box, which confused students, some of the student answer both questions, some of the students answer the top question only, some answer the last question only, some didn’t answer any question.
5.8 Few students related the information about currency in LILO back to what resources would be acceptable for their topic.
5.9 Students often wrote responses like “the book would be helpful because it is a good resources.” It seems as though asking them “why” is not prompting them clearly enough to integrate what they have learned in the prior LILO prompts about their resource.
5.10 - 5.11 These two prompts that appear on the same page are the only two prompts in the section on evaluating articles. Unlike most of the prompts in the other sections that relate directly to the student's research topic, these prompts ask for general information about the difference between scholarly journal articles and popular magazines articles. Almost all student responders were able to comprehend the difference between the two and most were able to go into some detail in their descriptions and reasoning.
5.12 Almost all student responders that were utilizing websites were able to locate the URL of the site, however, only about half of the students included the title in their response. Perhaps a discussion about the organization of the website above the prompt, including location of the title at the top of the page for most web browsers, would yield all of the desired information for the prompt.
5.13-6. Students’ answers varied wildly in this section as far as their plausibility, however, were generally much better than in the previous sections. Nearly all students had more to say but it seemed that many were confused about judging the authority of a website. For example several student cited personal websites as unbiased because they were maintained by someone that had an advanced degree! The students were very trusting of those with degrees even if they did not have an institution or references to back them up.
5.17 This prompt was clearly stated as students generally responded in a suitable manner, answering the question and usually providing detailed reasons of why or why not. It is interesting to note that the students overwhelmingly responded that the website they were evaluating was unbiased and appropriate for use.
5.18 This question is an overall question, it is clear, most of the student can give some part of the right answer. Again, there are many questions clustered together, so separating the questions into a more manageable format would be helpful.
This exercise is highly valuable to the future academic librarian. Through this project, we have experience creating standard based rubrics and if we so desire we can create some in the future for whatever instructional units or projects we are engaged in. Rubrics can give a general standard for assessment that can be used by almost anyone, thus giving instructors and students guidelines for assessment and making assessment by a variety of people or groups comparable. Through creating rubrics from student’s responses, we can evaluate a student’s information searching capability. We can assess which students need help for information searching for research papers. Creating rubrics assessment is standardized in such a way that it can be easily quantified. Rubrics provide the librarian with an easy way to quantify the success of teaching methods. This in turn allows the librarian to highlight their perceived importance to superiors in the university. Also, rubrics provide a means to clearly integrate the ACRL standards. This adds to the value because it allows the librarian to incorporate the skills being tested into the national goals for information literacy.
1.
“Center for Enhancement of Teaching and
Learning in California State University, Fresno.” www.csufresno.edu
. Home page online. Available from http://www.csufresno.edu/cetl/assessment/UsingScoringRubrics.pdf
. Internet; Accessed 27 April 2006.
2. Pon, Terrance. "Professional Development/Performance Evaluation model: Evaluation tools for teacher-librarians." School Libraries in Canada 24, no. 3 (2005): 1-1. Library, Information Science & Technology Abstracts, EBSCOhost (accessed April 25, 2006).
3.
Pickett, Nancy. “Guidelines for Rubric
Development.” San Diego State University. Home page online. Available from http://edweb.sdsu.edu/triton/july/rubrics/
Rubric_Guidelines.html. Internet; Accessed 25 April 2006.
4. Oakleaf, M.J. Created on 3/23/06 12:27 PM, Last printed 3/30/06 12:10 PM
5. “Objectives for Information Literacy Instruction: A Model Statement for Academic Librarians.” Association of College and Research Libraries. Home page online. Available from http://www.ala.org/ala/acrl/acrlstandards/objectivesinformation.htm. Internet; Accessed 17 April 2006.