Title:

Creating a rubric assessment tool: mapping Information Literacy Standards and defining degrees of competency for an online Information Literacy assessment tool.

By

Joe Murphy

and

Diane Kunichika

LILO Rubrics Project

LIS 665

Spring 2006

Dr. Nahl

ABSTRACT

This paper reports on the creation of grading rubrics as an assessment technique used to measure the meeting of Information Literacy (IL) Standards in the context of an online IL tutorial. The authors, both graduate students in the Library and Information Science Program at the University of Hawaii at Manoa, created these rubrics as part of Dr. Nahl’s course, “Teaching Information Technology Literacy.” Criteria and levels of competency were created for one module of the LILO (Learning Information Literacy Online) online IL tutorial. Rubrics were created by mapping prompts to ACRL IL Standards, Performance Indicators and Learning Outcomes. Suggestions are offered for improvement of prompts.

Introduction

We created a set of scoring rubrics for application to Module Four of LILO (Learning Information Literacy Online), the University of Hawaii’s online IL tutorial. LILO’s purpose is to introduce students – primarily first-year college students, “to the skills needed to research and write about a topic at the college level,” (University of Hawaii, 2005). LILO consists of six modules; we addressed the fourth section titled “Searching.” This section helps students learn to find the resources, such as books, articles and web sites, needed to write a research paper. Our goal was to define four degrees of competency to be used as assessment tools for this module.

Rubrics are an authentic assessment method. Traditional methods such as quizzes and tests only measure students’ ability to regurgitate information taught in class, a workshop, or online tutorial. Authentic assessment however asks students to “demonstrate, in a meaningful way, what they know and are able to do…authentic assessment emphasizes the application and use of knowledge...includes the holistic performance of meaningful, complex tasks in challenging environments that involve contextualized problems…require higher levels of cognitive thinking such as problem solving and critical thinking,” (Montgomery, 2002).

“Scoring rubrics are explicit schemes for classifying products or behaviors into categories that vary along a continuum.” Rubrics can be used to evaluate a “broad range of subjects and activities,” such as e-portfolios, annotated bibliographies, oral presentations, and performances (Moskal, 2000).

A useful introduction to scoring rubrics from the California State University at Fresno, “Using Scoring Rubrics” can be found at http://www.csufresno.edu/cetl/assessment/UsingScoringRubrics.pdf.

Knight explains the importance of describing in detail the meeting of designated outcomes (Knight, 2006). Knight also shares a short, helpful list of the necessary facets scoring rubrics should include. Our process was reflective of this model.

Our process mirrored that recommended by Moskal who outlined the basic steps of creating rubrics in “Recommendations for developing classroom performance assessments and scoring rubrics,” (2003).

Moskal offers suggestions for developing rubrics in scoring student responses from assessment tools (ED 2003). Helping with defining the criteria and levels of competency, Moskal explains the importance of clearly delineating exactly what evidence will be needed to accurately evaluate if the required knowledge has been gained.

Methodology

Four degrees of competency were defined in relation to meeting defined criteria to be measured for each prompt

We analyzed each of the twelve prompts included in LILO module Four, Searching.

ACRL Information Literacy Competency Standard 2, “The information literate student accesses needed information effectively and efficiently” and Performance Indicators (PI) one, two, three, and five guided our defining the levels of competency.

We carefully analyzed what each prompt in LILO Module Four is asking for and what is necessary to accomplish the specified goals. Then we mapped each of the twelve prompts to ACRL Standards and PIs, starting with those PIs previously mapped by the LILO committee. We then further mapped the prompts to appropriate Learning Outcomes. When the provided outcomes failed to adequately meet the prompt, we amended existing outcomes or created new ones. Outcomes are the flexible aspect of the standards, and their adaptability allows for their comprehensive application.

Standard Two was the only standard identified as pertinent to this module. We chose the ACRL Standard 2 PIs that would best measure each prompt’s question. PI One, Two, Three, and Five from ACRL Standard 2 were found to be the most applicable.

From the Learning Outcomes, criteria were defined that included each prompt’s pertinent information. It is imperative that these criteria summarize the skills/knowledge sought after in each prompt, and be aligned with the identified outcomes.

Identified outcomes were also used to define the levels of competency on which the students’ meeting of the defined criteria will be judged. There were to be four levels of competency for each prompt, “beginning,” “proficient,” and “advanced.” The “beginning” level covering both minimal answers, and responses including no discernable answer. Keeping the identified outcomes in mind, we defined exactly what each level of competency for each prompt would consist of. We accomplished this by defining what a “beginning,” “proficient,” and “advanced” response for each prompt would include.

Examples of each level of competency for each prompt were identified (Table 1 - 12). We scoured the lists of example answers from real students to isolate example answers that fit exactly what was made clear in each level of competency as we had defined them. These examples had to be clearly illustrative so that graders/evaluators, even nonprofessionals, looking at these examples would unequivocally know exactly what each level of competency should look like.

<>            The prompts, defined criteria, degrees of competency, and ACRL standards, PIs and outcomes were charted on tables one-twelve.

Discussion

We created 12 rubrics, one for each LILO prompt in Module Four. The content of the module covers how to find books and articles, and how to locate these items in the library using call numbers and location information contained in the Voyager OPAC records. Databases, web sites, and web search engines are also covered. The questions or prompts were tailored to and attempted to reinforce the content covered – some more successfully than others (see Future Research section page 9).

The rubrics we created represent one of two types of scoring rubrics called Primary Trait Analysis. This type of rubric is created by “defining the primary trait and then developing a rubric for each trait” (California State University of Fresno, 2002). We used this definition to guide us in the creation of our rubrics. The rubrics criteria were specifically formulated to measure the particular requirement as worded in the prompt. If the prompt was to test use of the Voyager OPAC then Voyager was referred to in the rubric. If the prompt included several requirements, only by answering all the requirements would the student receive an “Advanced” level score.

On the other hand, a “Beginning” level 1 score was created to address confusion on the part of some students as to what exactly the prompt required as an answer. It also allowed for greater flexibility than the original three levels of competency. For example if a LOC code was required the student would get at least a score of One for only writing out the description of the codes and not the codes themselves.

We considered both the question itself and the outcome we had mapped the prompts to. Most students were basically successful in their responses, but by no means universally. Sometimes students exceeded the explicit expectations of the question showing an advanced level of learning. We tailored the four levels to these degrees of expectation of the answers to the scope of student responses.

An example of this is for LILO prompt 4.3 (Rubric Table 3):

Browse the periodical article databases available at your library using the links above. Which databases do you think will be best for searching for articles on your topic, and why?

There was some confusion about exactly what a periodical database was – is it EBSCO or is it ERIC. This is a confusion that not only the general public has but also that LIS students have. The distinction is not explained in the text of this module. We decided however to not give such answers as “EBSCO is the best one” full credit but only a “Beginning” level rating. Some respondents answered “internet or an internet web site” – clearly didn’t get the point at all. This is an example of how we aligned the point levels to the LILO user group.

An example that shows the other end of student abilities – exceeding the implicit instruction in the question – LILO 4.8 (Rubric Table 8):

Do a search in the Hawaii Voyager Catalog to find an item related to your topic. In which library (or libraries) is the item located?

Here, in examining the responses we found that several students responded by writing out the item title and the library(s) in which the item can be found. Though the question does not explicitly state that the student should write out the item’s bibliographic information, we decided that this was an implicit component and was thus required to get the “Advanced” level score.

The main critique we have of our rubrics is to improve them through tying the rubrics more closely to the stated ACRL outcomes – to include language from the mapped ACRL outcomes. If we did this Prompt 4.1 and 4.10 could be clustered under a single criterion of “Student uses effective and efficient approaches to accessing information using an information retrieval systems.” In one case the information retrieval system would be Voyager and in the second case it would be the Internet.

Following are the Rubrics we created

Future research: Suggestions for improving contents and prompts.

We found most of the prompts in Module Four to be well phrased and very pertinent to the subject matter. Here are a few problems we found and suggestions for improvement.

Problems with Content

We’ve identified the pages with problems by the heading on the top of the LILO page as well as the file name.

Periodical Index Databases (find-articles1.php)

There are a lot of new terms in this section:

Periodicals

periodical index

databases

indexes

full-text links.

Our suggestion for improvement is not necessarily to leave out the terms but to arrange them better. Group similar topics together and state that only one term will be used henceforth. It’s a good idea to make the student aware of the fact that there are different terms to describe the same thing but use only one term throughout LILO. Picking and sticking to the use of a particular phrase will optimize student learning of the concept being taught without having to remember – oh yes periodical index and database is the same thing as something I can use to find an article.

Searching EBSCO’s Academic Search Premier (find-articles4.php)

The term “field” in the sentence “Databases are composed of fields of information (author names, article titles, journal titles, keywords, date of publication, etc.) that can be searched.” The term “field” specifically alludes to databases and the actual technology behind databases –one that the user does not need to understand. The word category would be a better choice here.

Problems with Prompts

LILO Prompt 4.2.1 –

Describe in your own words what periodical article databases are and what they include. How are they different from the library catalog, and how are they different from search engines?

There are two questions being asked here and the second question introduces a topic not discussed until 2 sections later – search engines. This question would be better as two questions and that the term “search engine” should be exchanged for specific search engines – Google or Yahoo.

LILO Prompt 4.3.4 –

In which collection is the item located? (If it's in multiple libraries, pick the one closest to you.) Is the item located in a special collection like Reference, Hawaiian, Asian, Music, or Government Documents?

"Collections" is not explained either on the page where this prompt is found nor on the previous page. The goal for this section is to help the student read the record in Voyager for the item and to eventually be able to locate the item. These two questions need to merge into one question to help the student preserve the item found in Voyager and to be able to find the item using information on the “Location” line. Our suggestion is for the student to cut and paste that information and then give specifics about the location. Move these two prompts to another page, the “Finding the item on the shelf” (call-numbers6.php) page for instance.

LILO Prompt 4.4.1. –

Follow the link to one of the search engine links above and search for your topic. List the title and the URL of one promising Web page you find.

Use of term “URL,” which some students seem to be unfamiliar with. LILO needs to provide an example or explain what an URL is.

Miscellaneous Problems

There is a navigational problem as soon as you enter the Module (Searching). The sub-menu is completely missing from this page.

Outcomes changed/created.

We amended outcome 2.1.c to read, “The student investigates the scope, content, organization, and accessibility of IR systems,” so that the issue of access is addressed.

We removed PL 1 from prompt 4.2.3 because we felt it did not address what is being asked in the prompt. We felt it not to be a helpful PI here because the prompt does not ask about selecting “the most appropriate investigative methods” or selecting “information retrieval systems.” The information retrieval system to utilize is dictated to users, and the method is not what is being judged in this prompt.

We created learning outcome 2.3.e, “The student compares IR systems & selects the correct system,” because we felt that no existing outcome adequately fit the criteria for this prompt.

Deficiencies in Performance Indicators covered.

Some PIs were utilized many times, such as PI One “The information literate student selects the most appropriate investigative methods or information retrieval systems for accessing the needed information,” and Three “The information literate student retrieves information online or in person using a variety of methods.” While PI Two and Five were only used once and PI Four is not addressed at all, showing a deficiency in the breadth of PI’s covered in this module. From this information we can identify the PIs that still need to be addressed either via newly created prompts, or reformulating existing prompts.

The large amount of students whose answers reflect a misunderstanding of the information requested in prompt 4.2.2 “Browse the periodical article databases available at your library using the links above. Which databases do you think will be best for searching for articles on your topic, and why,” makes it clear that this prompt should be re-worded. This may be an example of using what we as librarians assume to be clear language, but which comes across as jargon and is misunderstood by users as articulated by Hutcherson (2004).

A common thread I noticed throughout many answers across this module’s prompts was the confusion between vendors and databases. For instance, EBSCO was often presented as an answer to such prompts as 4.2.2 that solicit selection of databases.

The abundance of blank or inapplicable answers throughout this module may be indicative of outcomes not being met as widely as possible.

There is no marked difference in the rate of blank answers between these prompts. Prompts 4.2.4 and 4.3.3 received slightly more blank answers than other prompts, about 134 as compared to 143 non-blank results. Based on this information and criteria, these prompts should be the first to receive revision.

Conclusion; The value of creating rubrics to me as a future academic librarian.

Creating these rubrics were of tremendous value for us as future academic librarians. Until we did this assignment we were not aware of this means to measurably assess students’ performance of complex tasks such as research papers, presentations, bibliographies. In the earlier reading for this course we were urged to consider active learning methods and that veering away from traditional assessment methods would follow, but the question that kept popping up for us was how could we measurably assess the product? The answer we now know is rubrics.

Another positive aspect of rubrics is that in their creation the instructor must develop criteria, specific descriptions of the criteria and apply a scale to those criteria. This means the instructor must develop very specific and measurable behaviors or standards to create the rubric. In the creation of our rubric we considered the ACRL standards –also integrated into the rubric itself – to integrate a university’s education requirement for example.

Rubrics’ flexibility is its biggest asset. Criteria designed for a particular product or behavior can be created. For any product - research paper, presentation, group activity; create the criteria for that product; create levels of performance with a numeric scale, and apply the rubric to the product.

Rubrics can be “revealed” to students, providing them with a much better understanding of the instructor’s standards and requirements. The criteria lay out all the expectations for performance. There are other questions as to what the instructor will consider an “A” paper – rubrics leave no room for such doubt – it is all laid bare for the student.

Another adaptability of rubrics is its holistic applicability – on the global scale to IL programs, or to be applied more narrowly to a primary trait analysis – applied to a particular behavior such as web site evaluation. The holistic type of rubric scoring has a larger impact on IL competence in that a standardized rubric could be created to apply to IL programs and used to determine if a university’s or college’s information literacy program is in-line with an accredited body’s requirements such as is the case with the University of Hawaii at Manoa and Western Association of Schools and Colleges (WASC).

At the April 27, 2006 WASC “gathering of ideas for the proposal” meeting one of the authors used the information gained on rubrics from this assignment to impress how rubrics can be utilized in assessing IL standards. Rubrics can fill the need for assessing the meeting of IL competencies.

We have come across important usages of rubrics time and again in our studies and readings, during work experience in Hamilton Library, and in the larger library community. During this section of our course we have noticed an increased ability to interact with and utilize rubrics. There have even been examples wherein we were able to explain the application and creation of rubrics in library and education settings to administrators.

This information has also helped in observing the creation of criteria based on desired outcomes, though not explicitly worded as such, in the creation of a position announcement for example.

Creating rubrics from student responses to research journal prompts in the LILO module four tutorial has also been valuable to us because it has given us experience-defining levels of competency based on learning outcomes to measure student responses.

The IL instruction we will be engaged in as academic librarians, and assessment thereof will be of better quality because of our experience creating rubrics for this project.

Works Cited

California State University at Fresno-CETL, (2002, May 15). Center for Enhancement of Teaching and Learning. Retrieved May 1, 2006, from California State University at Fresno-CETL Web site: http://www.csufresno.edu/cetl/assessment/UsingScoringRubrics.pdf

Hutcherson, Norman B. (July 2004).Library Jargon: Student Recognition of Terms and Concepts Used by Librarians in the Classroom. College and Research Libraries. 65(4), 349-354.

Knight, Lorrie A (2006).Using rubrics to assess information literacy. Reference services review. 34(1), 43-55.

Moskal, B (2003).Recommendations for developing classroom performance assessments and scoring rubrics. Practical assessment, research and evaluation. 8(14), 1-8.

Moskal, Barbara M. (June 2003).Developing classroom performance assessment and scoring rubrics: part I. EDO-TM-03-02. 4 pages.

Moskal, B. (2000).Scoring Rubrics Part I: What and When. Report: EDO-TM-00-01. 4 pages.

Pickett, Nancy, and Bernie Dodge. (2001) “Rubrics for Web Lessons.” Retrieved May 2, 2006, from Rubrics for Web Lessons Web site: http://wequest.sdsu.edu/rubrics/weblessons.htm.

University of Hawaii (2005, September 16). Welcome to LILO. Retrieved May 2, 2006, from Learning Information Literacy Online at UH Web site: http://www.hawaii.edu/lilo/guest/lilo.php.