LILO RUBRICS PROJECT, by Caitlin Nelson and Wanda Wong

I. INTRODUCTION

LILO (Learning Information Literacy Online) is a project of the UH Libraries Information Literacy Committee (UHLILC). It guides first year English 100 students through the research process using web and library resources. Students create an online Research Journal which documents their research of a topic over a period of time.

Our assignment is to take one module of the LILO site, map degrees of competency of information research skills with ACRL-IL competencies, analyze open-ended student responses in the Research journals, then create a rubric assessment tool.

Rubric assessment in information literacy and higher education

Universities and colleges nationwide aim to increase Information Literacy in their students by striving to develop their problem-solving and critical thinking skills, elements of higher education that are vital to lifelong learning. The trend is to involve students in their learning, requiring them to demonstrate what they know by completing meaningful, complex tasks simulating real-life environments. (Montgomery, 2002)

Authentic assessment is based on performance and focuses on clearly defined performance outcomes, a measurable range of performance factors and authentic student performance data. It is especially effective for assessment in complex and subjective areas. A formative type of assessment, it becomes an ongoing part of the whole teaching and learning process, resulting in evaluations that are more reliable, impartial, and pertinent to the objectives. (CSUF, 2002) The Rubric is an authentic assessment tool. Rubrics contain three basic features: “a stated objective (performance or behavior), a range to rate performance, and specific performance characteristics arranged in levels indicating the degree to which a standard has been met.” (CSUF, 2002)

Authentic assessment is a high motivator in information literacy. Using rubrics to create and assess learning increases student confidence by spelling out criteria in ways that students understand the outcomes that are expected and where they stand in achieving those outcomes. Grading on a rubric becomes less subjective and more even or equitable. Another motivator, student satisfaction, is achieved by enabling learners to demonstrate their newly acquired knowledge in a reality-based exercise, bringing relevance and meaning of the lesson to the student. (Jacobson and Xu, 2004)

Rubric design

The table created by Megan J. Oakleaf (Librarian for Instruction and Undergraduate Research, NCSU), contained in our class handouts, lays out a clear-cut arrangement of rubric facets. It contains simplified performance conditions, three levels of student execution (from beginning to exemplary) for each condition, and corresponding learning outcomes integrated with ACRL indicators.

From this example, we set up our own rubric to integrate ACRL/IL Standards and performance indicators, LILO prompts, a range of measurements, carefully examined evaluation criteria, and, to illustrate the range of student performances in meeting the criteria, we extracted samples of student products representative of each level.

II. PROCEDURES

LILO module: Search Strategies
LILO prompts analyzed: 3.1 – 3.11 (for full text see Appendix B)

ACRL IL Standards, Performance Indicators and Outcomes (ALA, 2005):
Standard One: The information literate student determines the extent of the information needed.
Performance Indicator 1: The information literate student defines and articulates the need for information.

Outcomes:
1.1.a "Confers with instructors, etc to identify a research topic or other information
need."
1.1.b "...Formulates questions based on the information need."
1.1.d "Defines or modifies the information need to achieve a manageable focus."
1.1.e “Identifies key concepts and terms that describe the information needed”

Standard Two: The information literate student accesses needed information effectively and efficiently.
Performance Indicator 2: The information literate student constructs and implements effectively-designed search strategies.

Outcomes:
2.2.b “Identifies keywords, synonyms, and related terms for the information needed”
2.2.d “Constructs a search strategy using appropriate commands for the information retrieval selected (e.g., Boolean operators, truncation, and proximity for search engines; internal organizers such as index for books)

2.2.d1 “Refines keywords by applying appropriate commands (e.g. abbreviation, truncation, phrase marking, etc) in order to create searchable terms.”
2.2.d2 “Constructs a search strategy using Boolean operators.”

Extracting content

• First, we matched the LILO prompts to the ACRL IL standards.
• Then, we grouped the prompts based on ACRL Standard, Performance Indicator, and Outcome, and made one rubric per Outcome. Where we did not find an Outcome that was precise enough for our needs, we created one.
• Next, we created a general evaluation criteria for that rubric, based on the ACRL Standard.
• Then, we went through the raw data and "graded" the responses on a relative scale from “beginner” to “advanced” – not yet giving actual number scores.
• Then, we created criteria for beginning (0-1), proficient (2), and advanced (3) that would describe the “grades” that we had given the responses previously. (so, something we decided was "beginner" was given a 0-1 and we said based on what; something we said was "excellent" was given a 3, based on what, etc)
• Finally, we entered those criteria in the rubrics, and retested them on new raw data. Once satisfied, these become our final rubric criteria.

Primary trait analysis

Primary trait analysis is a process we used in defining the degrees of competency for module 3: Search Strategies. We scored student progress by defining the performance outcomes from LILO prompts, then developing a rubric for each trait. Tying in the behaviors required to progress through the module with student responses, we developed a range of grading factors for each behavior. Separate scores will then be available for each of the traits. (Picket and Dodge, 2001)

For our LILO rubric, the scores for level zero (No Response) and level one (Beginner) are rated together on a range of (0-1) with 0.0 for no response or an irrelevant response, and a 1.0 for a response that is vague but relevant. Scores for levels two (Proficient) and three (Advanced) are equally rated with 2.0 points and 3.0 points, consecutively.

Attesting to the effectiveness of authentic assessment in complex and subjective areas, we found that approaching the LILO module by means of the rubric enabled us to determine the cognitive processes of the students in each of the prompts. From there we were able to focus on the objectives of the lesson and restructure the prompts and performance indicators.

III. THE RUBRICS

Our rubrics are presented in Appendix A

IV. DISCUSSION

By going through the LILO tutorial and completing the LILO prompts, we were able to mimic the process by which college freshmen follow the lessons. Then, by examining the Journal responses to each of the prompts, we were able to follow the cognitive trail of individual learners.

We found that some responses revealed learners who were able to understand the points of the lesson on each page. They were able to not only follow the prompts and contribute good answers but, in some cases, go over the minimum expectations.

There were others who gave us the impression that they understood in general terms what was being taught, but were not able to provide concrete evidence that they were comprehending anything, although, in some cases, they seemed catch on in later parts of the lesson.

We scrutinized our own experience with the prompts and matched the Journal responses to the questions. We felt we had to tailor the module to bring forth better understanding of the lessons and elucidate better answers. We did this basically by changing terms and phrasing so that the questions were more pertinent to the thought process. We also changed the sequence of prompts to make the module more logical and productive in gathering search terms for future use.

By actually going through the module ourselves, we were able to gain invaluable insight into LILO and into the rubric creations process itself. We believe that we created rubrics that will accurately and effectively assess the Research Journal data, and that the changes we made to LILO will facilitate the students’ learning process. At this stage we are confident in our work, but in future stages the rubrics will have to be tested on new data, and with a series of different graders; only then can they be revised and updated to be even more useful.

V. FUTURE RESEARCH

In order to create our rubrics, we had to become very familiar with the LILO module “Search Strategies” and the accompanying Journal prompts. During the rubric creation process, we made several major changes to both the prompts and the flow of the module to create what we think is a more user-friendly, clear and concise unit. The changes we made are as follows:

First of all, we suggest a change to the order of the module. Currently, the prompts lead the student through the process of : (1) identifying a broad topic; (2) narrowing that topic; (3) “brainstorming for search terms”; (4) identifying synonyms for those search terms, and alternate spellings of all words; (5) using LILO application “Keyword Builder” to create Boolean search strings; and finally (6) using truncation and/or phrase marking.

This order, by finishing on “truncation” and “phrase marking,” leaves the student hanging by not allowing them to utilize all the skills they learned in this module in building actual search strings. Therefore, we have reorganized the module to be: (1) identifying a broad topic; (2) narrowing that topic; (3) “brainstorming for search terms”; (4) identifying synonyms for those search terms, and alternate spellings of all words; (5) using truncation and/or phrase marking; and finally (6) using LILO application “Keyword Builder” to create Boolean search strings – basically switching the last two sections.

We also suggest modification of some of the terms used and the wording of the prompts. We suggest revision of the section “Selecting Search Terms” to replace the word “concept” with “searchable terms.” At this point in the module, the student should have moved away from creating more/different concepts, and should be focused on taking the concepts he/she already has and thinking up synonyms and variations on those words.

Therefore, original prompt 3.4 (“Look at your questions and break down your main idea into 2-3 component concepts. List them here:”) and 3.5 (Brainstorm synonyms and closely related terms for each of the concepts related to your topic and list them here) become:

3.4 Look at your questions and break down your main ideas into 4-6 searchable terms.
3.5 List any appropriate synonyms for each of your searchable terms.

Our other changes mainly have to do with the Keyword Builder (hereafter KB) section of the module. Originally there were three questions having to do with the KB:

3.7 Copy the existing sample search string from the Keyword Builder and paste it here.
3.8 Select two terms from each concept listed above and enter them into the Keyword Builder. Paste the search string here.
3.9 Enter your own search terms into Keyword Builder and paste the search string here.

However, the first one (3.7) is merely a cut-and-paste exercise, and clearly caused confusion among the students; some of the students followed directions and cut-and-pasted exactly as they were told, some started by entering their own search terms to the KB, and others did neither – either entering nothing, or something else entirely.

After looking at the data and reviewing the module, we decided to cut this question for two reasons. One, the data is too disparate to be useful; clearly not enough students understand the directions properly. Two, we feel that the question is not useful in teaching the students to use the KB, and only serves to confuse them. Instead, we have suggested several revisions to questions 3.8 and 3.9.

Question 3.8 – an exercise designed to give the students practice using the Keyword Builder – should be removed from the Research Journal entirely, as the answers entered have nothing to do with the students’ actual research topic. Instead we propose a modified section where the students enter the data as shown on the screen (“Select two terms from each concept listed above and enter them into the Keyword Builder.”) and on the next screen are shown proper examples of what the search should have looked like. For example:

First page: Enter two terms from each concept to the KB:

1. films movies, motion pictures, videos
2. criminals convicts, lawbreakers, felons
3. vehicles cars, planes, motorcycles, bicycles

Second page: Your Keyword Builder search should look something like:
( film OR videos ) AND ( automobiles OR cars ) and (crime OR criminals OR perpetrators)
( films OR movies ) AND ( vehicles OR cars )
( films OR movies ) AND ( crime OR criminals )

Did your search look like this? If not, go back and try again!

Finally, Question 3.9 has been revised (and renumbered) to say:

3.10 “Select keywords from your list of searchable terms to create two new searches using the Keyword Builder.”

This page should be the culmination of the “Search Strategies” module – the students should be able use all their search strategy-building skills to put together usable search strings using their own keywords. To this end, it would be useful to call up all the keyword data they have previously input into their Research Journal, and display it here. Therefore, the new page would be something like:

Here is the list of the keywords, synonyms, phrases, abbreviations, and truncated words / you have input so far:
[LIST DISPLAYED HERE]
Select keywords from your list of searchable terms to create two new searches using the Keyword Builder. [KB linked here] Paste these search strings into your Research Journal. Now you can use these search strings later!

These changes are shown in the “comments” section of each rubric. We hope that by making these changes, we have made this LILO module more coherent and useful for future students, and that we have created prompts that are clear and produce more relevant data for future librarians and professors.

VI. CONCLUSION

Value for future academic librarians

The responsibility for teaching information literacy in academic libraries is growing exponentially as information technology takes a greater hold on student learning. A substantial portion of the dialogue for the design and implementation of literacy programs in colleges and universities is being carried out by instruction librarians.

Incorporating standards-based methodology such as the ACRL Information Literacy Competency Standards and the ARCS Model for Motivational Design into evaluating an established tutorial like the UH LILO is invaluable for enabling future academic librarians to contribute to student learning in the general curriculum. The experience prepares MLISc graduates for the challenges of real-world information technology literacy instruction for new generations of undergraduates.

Whether we assess a learning tool already in place, design our own teaching modules, or integrate literacy workshops into classroom and program curricula, our use of authentic assessment and the tool of rubrics will move our individual teaching and learning aspirations towards higher levels of accountability.

Program assessment

According to Picket and Dodge (2001), rubrics for instruction program assessment allow evaluators “to efficiently focus on specific learning objectives while reviewing complex student products…. without getting bogged down in the details” Clearly defined expectations and processes are delineated for closer examination and deeper understanding.

When used to assess a multifaceted objective such as an internet tutorial, the rubric does its best job as a powerful tool for clarifying judgments or valuations in response to complex, layered, or subjective criteria. It gathers exceptional evidence in matching performance with expectations and standards.

In addition, rubrics to assess program competence invite communication and collaboration between students, librarians, faculty, technology professionals, and administrators. Teaching, learning, and program assessment become a shared cultural experience. What better way to promote lifelong learning in ourselves, our students and our community.

References

American Library Association (ALA). 2005. "Information Literacy Competency Standards for Higher Education." Internet. Accessed 25 April 2006; http://www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htm#stan

Association of College and Research Libraries. 2003. Assessment Issues. Information Literacy. Internet. Accessed 25 April 2006; http://www.ala.org/ala/acrl/acrlissues/acrlinfolit/infolitresources/infolitassess/assessmentissues.htm

California State University at Fresno (CSUF). 2002. Teaching, Learning and Technology: “Using Scoring Rubrics.” Internet. Accessed April 25, 2006. http://www.csufresno.edu/cetl/assessment/UsingScoringRubrics.pdf

Jacobson, Trudi E., and Lijuan Xu. Motivating Students in Information Literacy Classes. 2004. New York: Neal-Schuman Publisher, Inc.

Picket, Nancy, and Bernie Dodge. 2001. “Rubrics for Web Lessons.” Internet. Accessed April 25, 2006; http://webquest.sdsu.edu/rubrics/weblessons.htm

Here we present the series of rubrics we created for LILO module “Search Strategies.” There are six rubrics in total; each one shows the corresponding LILO prompt(s) and ACRL IL Standard, Performance Indicator, and Outcome. The general Evaluation Criteria are presented at the left, and each level criteria are presented in order. Below that we have entered comments, if we had any changes for example, and quotations from actual responses that are examples of the matching level.

Original Prompts

3.1 What is your broad topic?
3.2 Subdivide your broad topic by identifying 2–3 more narrow and focused topics, using strategies outlined above
3.3 What questions do you want to ask about your topic?
3.4 Look at your questions and break down your main idea into 2–3 main concepts, and list them here:
3.5 List any appropriate synonyms for each of your concepts.
3.6 List any alternative spellings, abbreviations, or multiple word endings that apply to your concept terms or synonyms.
3.7 Copy the existing sample search string from the Keyword Builder and paste it here:
3.8 Select two terms from each concept listed above and enter them into the Keyword Builder. Paste the search string here.
3.9 Enter your own search terms into Keyword Builder and paste the search string here:
3.10 Are there any terms in your search string that you might use a truncation symbol with? List them here:
3.11 Are there any phrases related to your topic that you might need to use quotations marks with? List them here:

Revised Prompts:

3.1 What is your broad topic?
3.2 Divide your broad topic by identifying 2-3 more narrow and focused topics, using strategies outlined above.
3.3 What questions do you want to ask about your topic?
3.4 Look at your questions and break down your main ideas into 4-6 searchable terms.
3.5 List any appropriate synonyms for each of your searchable terms.
3.6 Enter any alternative spellings, abbreviations, or multiple word endings that apply to your concept terms or synonyms. If there are none, enter N/A.
3.7 Enter any terms from your lists of keywords that you could use a truncation symbol (* or ?) with. If there are none, enter N/A.
3.8 Enter any phrases related to your topic that you would need to use quotation marks with. If there are none, enter N/A.
3.9 Enter your own search terms into the Keyword Builder and paste the search string here.
3.10 Select keywords from your list of searchable terms to create two new searches using the Keyword Builder.

2006 © Caitlin Nelson and Wanda Wong, UHM/LIS