• Faculty Questionnaire
• Student Demographics and Practices
• OASIS ARCS Evaluation

Of the 103 faculty who received questionnaires, 23 returned the questionnaires by the due date (22.3%). However, two teachers returned blank surveys and two were incomplete because those teachers did not utilize computers or the Internet in their class. During informal encounters with teachers later, the investigator learned that many teachers did not return their surveys for the same reason. However, the investigator was attempting to obtain all faculty opinions, not just feedback from those who utilize computers in the classroom. This was an indication that the investigator needed to be clearer in the directions that were given with the questionnaire. Those teachers and other faculty who did return the questionnaires were from various subject areas as seen in Table 1. Teachers ranged in experience from first year teachers to over 30 years teaching. The average number of years was 11 years.

Table 1. Faculty Responses by Area (n=23)

Responses	Area
3	English
2	Special Education
2	Math
3	Foreign Language
3	Science
3	History
5	Other (Art, Photography/Yearbook, Business Communication, Health Occupation, Counseling/Physical Education)
2	Blank

Back to top

Table 2 shows that all 19 respondents with completed surveys felt that standardized guidelines for Web resources would be valuable for teachers and students. However, only six of those respondents provide criteria for Web resources. One respondent even wrote that she or he "depends on English classes to do that." As information professionals, school librarians could serve in that role, but only eight of the respondents collaborated with the librarians on research projects.

Table 2. Faculty Responses to Questionnaire (n=19)

No.	Statement	Responses
1.	I assign research projects in my class.	19
2.	I provide guidelines for research projects such as length and number of resources.	17
3.	I require students to submit a bibliography, or similar listing, for their projects.	16
4.	I provide a bibliographic format for students to use.	11
5.	I explain resources that I will accept as valid references for research papers.	14
6a.	I allow students to use electronic resources such as …CD-ROM.	16
6b.	… television shows.	12
6c.	… videotapes media.	15
6d.	… audio tapes.	13
6e.	… World Wide Web.	19
7.	I provide guidelines as to how to cite books, journals, and encyclopedia entries.	12
8.	I provide guidelines as to how to cite electronic references (e.g. Internet, Web, CD-ROM).	11
9.	I check every reference that students use in their reports.	3
10.	I generally accept all Web resources as valid.	6
11.	I provide criteria for valid and reliable World Wide Web information.	6
12.	I provide Internet resources for my students to supplement learning and instruction.	10
13.	I have time to research World Wide Web sites for students.	4
14.	I feel that standardized guidelines for Web resources would be valuable for both teachers and students.	19
15.	I collaborate with the school librarian on research projects.	8
16.	I give my students library time for research projects.	13

When it came to multimedia resources utilized in research projects, of the five media listed, the World Wide Web was the most accepted followed by CD-ROM (16), videotapes (15), audio (13), and television media (12). Ten teachers even provided Internet resources for their students. When asked about student project research time, teachers indicated that they gave an average of 2.8 weeks for a research project and some (13) even gave some library research time. Also, when asked if they felt that students had an advantage if they have the World Wide Web to use as a research tool, majority of the 14 teachers who responded to the question said, "Yes" as seen in Table 4.

Back to top

Table 4. Summary of Faculty reponse to Question: Do you feel that students have an advantage if they have access to the World Wide Web.

Faculty Number	Response
1	Yes, makes many additional options available. However, also increases possibility that student will simply use the verbiage directly from the net and skip the analysis-synthesis phase. This is a big concern.
3	Yes. Some information are more current.
4	Yes, it provides many more sources, it will also teach them how to shift through things to find out the truth.
6	The instantaneous information often leads students to view other sites. It shows students that the vast area of books, mags, etc. available. It can also help to verify - point - student is unsure of.
9	Yes, - the access is invaluable concerning availability of information
10	Yes, because the information on certain topics may be difficult to find elsewhere.
15	Yes, access to data - but warning as to content, reliability.
16	Yes, it is a great library
17	If the kinds of projects were research driven, then the question would be much more valid. Projects are more activity based, data collecting.
18	Yes, but it's hard to determine when they plagiarize and not use their own words.
19	Yes and No. Yes, can scan more information faster, useful in selecting topics. No, frequently, they forget that valuable information can also be found in journals, reference books, etc.
20	Yes, but only if they use it properly and cite it properly.
21	Yes, I assign research projects on what otherwise would be too specialized and obscure a topic for a high school paper.
23	Yes, if they are competent enough to take advantage of this resource.

Overall, the faculty responses to the questionnaire gave the investigator good insight as to how teachers are viewing the World Wide Web in regards to research projects. The investigator felt that the support and validation for the project from the faculty was very important in the development of the instrument. Also, the results indicated that specific effort needs to be put into development of such an instrument by an outside party other than the classroom teacher because he or she has little time to do it him or herself.

Back to top

The students who participated as part of the small group were enrolled in a history, a science, or an English class. The student groups were selected so that no student would be enrolled in the participating classes during different periods. As seen in Table 6, majority of the students were in the 10th grade.

Table 6. Student Demographics

Grade	English	History	Science	Total	Percent (%)
Blank	0	1	3	4	5.7
9th	0	1	0	1	1.4
10th	0	15	22	37	52.9
11th	18	0	7	25	35.7
12th	0	1	2	3	4.3
All	18	18	34	70	100

In regards to computer use in home and at school, majority of the students (88.6%) owned a computer at home and 84.3% used computers to do their homework. Ninety-four percent of the students had access to a computer either at home, at school, or elsewhere. At least 80% of the students had Internet access from home. Surprisingly, 55.7% of the students indicated that they had access to the Internet from school though only 21.4% of the students in the study have an Internet Driver's License. At Roosevelt High School, all students were required to earn an Internet Driver's License to utilize the Internet. During the 1999-2000 school year, all 9th graders were required to earn an Internet Driver's License through their English courses, but the license was optional for upperclassmen. The Internet Driver's License, developed by Cathy Watson at Kailua Intermediate School in response to the Technology Literacy Challenge issued by President Clinton and Vice President Gore, is a module that explains the basics of Internet usage and etiquette. Students work through the module then have to pass a quiz to earn the license. Since more than half of the students in the study indicated that they have access to the Internet in school, some teachers may be allowing the students to utilize the Internet without the license.

A summary of the content portion of the student survey is seen in Table 7. Most students agree that they utilize the Web to do research (x-2.07) and are aware of aspects of research regarding the Web such as how to search (x=1.7) and how to list resources (x=1.93). The only two statements with which students disagreed were Statements 6 and 10. Statement 6 stated that they learned how to search the Web in school (x=3.74).

Back to top

Table 7. Results of Student Survey

Figure 1a-d shows the difference in opinion for four of the open-ended questions in the student survey. As seen in Figure 1a, there was not great difference between those who go to the library first (41%) as opposed to those who go to the Web first (47%). During a discussion after the surveys were submitted, some students clarified that they went to the library first to specifically use the computers and the World Wide Web, not look for information on the library shelves. However, on their survey, they wrote library because that was where they physically went. Because the surveys were anonymous, the investigator could not adjust for this technicality. Therefore, the statistic for the Web is most likely higher than seen here.

There was not a great difference between those who prefer the Web (51%) over the library (46%) or vice-versa (Figure 1b). During discussion, several students said that they preferred the library because it was readily available whereas they did not have access to the Internet to use the World Wide Web. If they had access, their preference would change. However, 52% of the surveyed students, as seen in Figure 1c, did feel that resources from the Web were better than those from print materials such as books, magazines, and encyclopedias. Majority of the students (79%) also felt that the Web gave them an advantage when doing research, as seen in Figure 1d, mainly because of graphics and audio media could also be obtained as well as print information. Other reasons were that it was the fastest and most efficient way to look up resources because they had such short periods of time to do a research project. Those who felt that the Web did not give them an advantage maintained that information from the Web was not trustworthy and reliable. They wrote that they would rather use a resource such as books that had an editing and publishing body that was responsible for the information.

Back to top

The last question of the survey asked students to list criteria that they used to evaluate Web sites as potential resources for their research projects. Forty-eight students responded to this question with various answers, half of which were related to graphics. Of those responses, about one-fourth of students stated that they did not know. After the completion of the project and use of OASIS, 87.6% of the students were able to correctly state the keywords to OASIS to guide their evaluation of a Web resource.

Back to top

Subject Matter Expert

The SME was involved with the development of the ideas and background research for OASIS. Because of this involvement, there were few comments regarding the final instrument. Some written and oral comments about OASIS included:

- It will be a useful tool for students.

- I like the fact that it is brief.

- The only item I felt needed some re-wording was the item on time span. I am not sure what you were trying to ask??

- On the question about using the acronym OASIS, it may mean something to teachers but students would not be familiar with it.

Overall, the SME provided direction for the five key points by guiding the background research during development. Her insights into research skills and information literacy helped define the keywords used in the final instrument.

Back to top

Knowledgeable Learners

Of the twelve knowledgeable learners who participated in the study, three were librarians, five were technology coordinators, and four were teachers. All have been teaching for at least eight years. Half of the participants had over twenty years of teaching experience. The KNL taught at various school levels throughout the state, three at the high school, one at the middle school, and eight at the elementary level. At their respective schools, only seven had some form of guidelines for students to use when using the World Wide Web.

As seen in Table 8, KNL gave substantial written feedback on OASIS. Areas that they felt were good were Source and Information. Areas that still need development are Objectivity, Accuracy, and Span. OASIS was designed at a high school level. However, because most of the KNL were elementary teachers, there were also specific comments about modifying OASIS for use at the elementary level. Comments about the modification for elementary students will be left out of the following discussion.

Under Objectivity and Accuracy, suggestions were given to expand the questions to include a comparison exercise to help students understand the difference. However, for the purpose of this project, this step was not included into the instrument, but may be incorporated into an instructional module if developed in the future. Other comments included adding more clarification questions and expanding the one-page instrument into an instructional module comparable to the Internet Driver's License module.

The Span criterion was the area that caused the most confusion and concern. Because of a typographical error, the first question was awkward and did not make sense. Revision of the question was needed to make the criterion clearer to both students and KNL. Otherwise, the KNL commented that touching upon both the updated nature of a Web site and the time reference of a research project was good.

Table 8. Comments from Knowledgeable Learners

Criterion or Question	Comments
Objectivity	- They need to establish that they can recognize the difference tween these two. Criteria about facts? - How will elementary students be able to determine if information is fact or opinion. Elementary students need more specific prompts. - A little difficult for students to ascertain whether it is fact or opinion without being able to compare with another source.- We do bring this to the student's attention, but at the elementary level, we need to give examples of what is a fact or opinion.- Concise- Good points for students to consider.- Is that possible to tell (by most students)
Accuracy	- I think these are covered well.- see above- Who is the someone else?- How does the information compare with at least three other sources I've used?- Again, I cover this, but I need to discuss with the students the vocabulary of valid and reliable, bringing in examples.- Some students may have a problem with the meaning of "valid"- How will they verify the information?- good there
Source	- well questioned- see above- What do you mean by other references?- I introduce all of this, so the students do know where to look and what to look for, but at the elementary level, some will "catch on" or remember and some don't care.- Good- Excellent questions.- excellent and clear
Information	- good, especially for higher levels- see above- Again, assuming they have enough information to compare - however, your premise is that they go to a web site first. - I do talk to the students about persuasive writing for the commercial sites. After reading this, I'll start talking about other purposes of the information. We do talk about including library sources as a must.- Good- Helps focus on content.- Not sure how one could answer the second question without exhaustive searching already completed.
Span	- phaseology(sp) of the first is a little difficult to follow- The first criterion sounds a little confusing. see above- First question is awkward - may want to rewrite Dated and updated may be confusing to students- First question not too clear. I'm not in agreement with your use of the term Span. - Touched on this in terms of really outdated materials, but the way you put it here also talks about updating information. That's good.- Questions are a little confusing - Is the date of publication current enough? Is the information dated (not current) or is it updated (recently worked on)?- I think there is a typographical error here.- Is the time span right of when this was created right for my topic? Doesn't make sense!
OASIS Confusion	- I prefer not to have acronyms, but to work for comprehensiveness without trying to fit a model. My major problem with everything above is that there is no explanatory material for them to base their decision on. There needs to be MUCH more - I don't know what it stands for either.- I've never heard of it either... I plan to use a simpler rating system for my students since my intention isn't really to critique websites that will be used for research papers, but just to get them to see what's out there and determine w- I am sure with lots of practice and teacher reviews the students can understand what is wanted. At our school, we are at the stage where children are trying to do a good citation of a Web page.(Bibliography)- I think this is a great tool and very appropriate for intermediate and high school kids. It would be great if you can "translate" this tool into language that elementary school kids can use. In this form, elementary teachers, and parents- For elementary, where this should be first implemented, it might be better use a graphic of a hand and relate the 5 parts to the 5 fingers of a hand. I don't think the older students would be put off by this. You could make it more appeal- I think that if you use an acronym, it should be one that has relevance to the purpose - that would help the students remember. For example, WEBRIGHT, or something like that. Also, its easier if you give them "yes" "no" areas to check/cir- I would prefer something simpler like Web Check or similar.- OASIS is fine. Again, I would have to make sure the students understand what an oasis is so they can assossiate.- Acronyms make it easy to remember things. According to national technology standards, the evaluation of electronic information sources should include: Accuracy, Relevance, Appropriateness, Comprehensiveness, and (Identification of) Bias.- I immediately questioned the name. I think DOE employees would find it confusing. If I were you, I would select another name.- Yes, I think it would be confusing even if the students have't heard of the other one. Once they DID then they would wonder if it was the same. Right away, I thought maybe OASIS created this.
Additional Comments	- Too brief- Thank you for your efforts. It should facilitate things for all of us. - see additional comments above- no- I think there should be some comparing of information from a number of different sources.- For elementary level, there would have to be a lot of prep on the guidelines before they can actually use it.- This is suitable for elementary as well as high school.- Seems very good. Only comment would be in allowing multiple input for school type (middle and High or elementary and middle, etc.)

Back to top

Knowledgeable Learners and Students

Both the KNL and students completed the OASIS Evaluation Form. The evaluation was based on the Keller's ARCS method where Attention, Relevance, Confidence, and Satisfaction are measured. Items 1 and 2 were in regards to Attention, 3 and 4 to Relevance, 5, 6, and 7 to Confidence, and 8 and 9 to Satisfaction. Results can be seen in Figure 2. Based upon a five-point Likert scale, both groups agreed with all statements with the exception of Item 5 for students. Because feedback was collected anonymously over the Web, the investigator was not able to contact the KNL for comments about items that the investigator wanted clarified.

Figure 3. Knowledgeable Learners and Student ARCS Evaluation Results

Student opinions were not as positive as the KNL in all areas except for Items 1 and 4. Item 1was in regards to Attention and focused on the key graphic that was designed for OASIS. After talking to several students, the investigator learned that students would rather have a picture of an oasis, e.g. a palm tree by some water surrounded completely by sand. They liked the picture of the key in reference to OASIS as an evaluation key or method, but it did not gain their attention as much as a picture of an oasis would. The KNL were also neutral about the graphic, but did not suggest on any possible alternatives. Item 4 was in regards to Relevance and was about understanding why Web resources need to be evaluated. Both students and the KNL did not offer comments on this item.

One area in which the students disagreed was Item 5 in regards to Confidence. The statement was about the ease of remembering the keywords to OASIS. Though students were not as confident about this statement, after the completion of the project and use of OASIS, 80.5% of the students were able to correctly state the keywords to OASIS to guide their evaluation of a Web resource.

Back to top