METHODOLOGY
System
The system into which this evaluation tool was incorporated was Roosevelt High School. Roosevelt High School is the Engineering and Technology Learning Center of the HIDOE. The advancement of technology in education is one of the goals of the school. One particular goal is to increase critical thinking skills and cognitive evaluation of the World Wide Web. The principal and administration, other teachers and faculty, librarians, and parents directly affect the progress and development of ideas by providing support to the students in their endeavors.
Figure 1. System Chart
The sub-system consisted of students and teachers in three distinct classrooms in three different subject areas. The students were the main users of the evaluation tool developed in this project. Teacher acceptance of the evaluation project was vital to the success or failure of the project. Several components of the sub-system included student knowledge of the Web and its features, computer and Web availability to students, and instructional time in the classroom. Student knowledge of certain features of Web pages and Web sites were important to effective utilization of the evaluation tool. Computer and Web availability to individual students was also a factor in motivation. If students did not use the Web as a resource, then there was little motivation to utilize the evaluation tool. Instructional time that was needed to introduce and explain the tool was a large factor for teachers who may have been apprehensive about allowing the use of the Web in research projects. The teachers' opinions and lack of support for the project may influence student opinions and motivation.
The suprasystem is the HIDOE. The HIDOE is divided into two main divisions, one of which deals with the incorporation of technology into curriculum. Effective and efficient use of technology to enhance learning was the overall goal of this division. It was believed that this project could add to that particular goal. Statewide administrators and professional development teams and parents strongly influence the use of curriculum and instructional practices in the public school system. Also included in the supra-system were the technology coordinators who work with the teachers, and students to provide access to the Web and other computer applications for completion of student research projects.
The project also involved subject matter experts who were individuals in the University of Hawaii, hereafter referred to as UH. Administrators and project teams in the HIDOE often consult UH professors and faculty for their scholarly opinion. The UH College of Education also provides instruction and training for most teachers who were in the system or pre-service teachers who would be entering the system.
As a production project to create a comprehensive evaluation tool for use by students, this project was conducted in five phases: (a) identification of evaluation criteria, (b) development of evaluation tool and procedure, (c) field testing of prototype, (d) revision of evaluation tool, and (e) presentation for acceptance by the State Department of Education.
Phase one dealt with the identification of criteria for evaluation purposes in all media. The initial source for this phase was the 1997 study conducted by (Wilkinson et al., 1997) that listed 125 indicators of quality. Also included in this phase was the compilation of 66 known evaluation tools for Web use. The list of print and electronic resources utilized can be found in the Appendix A. Consultation with a Subject Matter Expert, hereafter referred to as SME, was also conducted.
After researching, compiling, consulting, and editing, a draft of an evaluation tool for students was developed during phase two. This tool, print or electronic, was no more than two pages long in consideration of student needs. The evaluation tool, entitled OASIS (see Appendix B), was distributed to the SME again, undergoing further revision to only one page.
The project tested the evaluation tool at a local high school in its third phase. Three classes with different subjects-science, language arts, and history-were used since it was in these three subject areas where research projects were most prevalent. Student demographics were gathered at this point along with feedback about the evaluation tool. As a result, the evaluation tool was revised again (phase four).
Phase five involved preparing the evaluation tool for presentation to the State Department of Education for adoption.
There were five distinct groups of participants in the project. They were the faculty at Roosevelt High School, the SME, knowledgeable learners, one-on-one learners, and student groups.
Faculty initially received a faculty questionnaire to assess the need for and participation in the project. The SME, knowledgeable learners, and student groups were given OASIS and OASIS Evaluation accompanied by an explanation of the project. Students also received a student demographic survey. Refer to Appendices B-E for samples of the faculty questionnaire, student survey, OASIS, and the OASIS Evaluation Form.
Tasks of the Investigator
The investigator conducted the research on existing forms of evaluation and criteria, made contact and scheduled meetings with all subject matter experts as well as technology coordinators, librarians, and teachers, visited classrooms and labs for introductions and explanations of the project and evaluation tool, collected all data involved, and wrote the formative evaluation of the project. Also, the investigator prepared the final revised project for presentation to the State Department of Education for consideration of adoption in the Office of Technology and Information Dissemination.
Faculty
The faculty was asked to complete the faculty questionnaire (Appendix B) and comment upon the World Wide Web in education. The Web resource evaluation method had not yet been developed at the time that the faculty was surveyed, therefore, faculty did not see OASIS. Faculty feedback was extremely positive and encouraging.
Subject Matter Expert
The SME was asked to review the research behind OASIS and the final OASIS instrument itself. Her comments regarding the research behind OASIS will not be presented here. There were not many comments about the final product. However, her constructive comments and encouragement enabled the investigator to find models for development of the final instrument.
Knowledgeable Learners
The knowledgeable learners, hereafter referred to as KNL, were asked to review the OASIS instrument and complete the OASIS Evaluation Form via the Web. They were asked to comment specifically on the comprehensiveness of the questions in each section and about the use of the OASIS acronym that may cause confusion in State of Hawaii Department of Education employees. The reviewers provided substantial written feedback and many positive comments.
One-on-One
The student was asked to review the student demographic survey, OASIS, and the OASIS Evaluation. The students looked specifically at the wording in and clarity of the directions and questions within each section. Additionally, he commented upon the aesthetic design of the page for high school students.
Student Groups
The student groups were asked to utilize OASIS in a research project, give written or verbal comments for improvement, and complete the OASIS Evaluation form.
Faculty
The faculty to whom the faculty questionnaire was distributed was full time or part time teachers, counselors, or administrative personnel at Roosevelt High School. A project description was given to and permission was obtained from the school principal prior to distributing the questionnaire. A letter of introduction and explanation of the project accompanied the questionnaire along with the investigator's and school liaison phone and e-mail contact information. Arrangements were made to have the questionnaires dropped-off at the school library one week from the date of distribution. A request for volunteers to participate in the project was also made at the end of the questionnaire. In addition to completing the questionnaire, faculty was also encouraged to write comments about their feelings about the role of the World Wide Web in education.
The investigator was present in the library during after school hours every day that week to be available to faculty who may have had questions. At the end of the one-week period, the questionnaires were collected from the head librarian. The written and verbal comments from the faculty were invaluable. All teachers supported the project and many voiced concerns over potential misinformation and lack of criteria for information on the World Wide Web.
Twelve teachers volunteered their classes to participate in the development of the Web resource evaluation tool. Due to the scope of the project and available manpower, only three were selected to participate in the project. The investigator met with all three teachers to discuss issues and stages of the project that would involve their classes. A schedule was planned for all three classes where the teachers would distribute a student demographic survey, the investigator would meet with the students to explain the project and distribute OASIS, and the investigator would return one week later to collect comments and answer any other questions.
The SME who reviewed OASIS was an Associate Professor in Computer Information Sciences at the University of Hawai`i at Manoa. She is an active librarian, researcher, and instructor in the areas of information research skills and evaluation methods. She also serves as the Chair of the Library and Information Sciences Department.
The investigator approached the SME a year ago about the need for an evaluation tool for high school students. Arrangements were made to have the SME serve as a reviewer primarily over e-mail due to scheduling constraints. The SME reviewed preliminary research for and the final one-page OASIS via the World Wide Web. The feedback received from the SME provided good insight to the field of research and academic approaches to teaching research skills. Also, she provided invaluable contacts to other information educators and professionals and leads to Web information criteria for librarians and teachers.
The KNL were derived from two groups. The investigator and SME identified two groups of educators as those with high motivations for development of OASIS-the Tech Cadre and school librarians. The Tech Cadre was a group of educators who shared a belief in the pursuit of excellence in the field of science and technology. School librarians possessed specialized skills and training that enabled them to work closely with teachers to develop effective research and evaluation skills in students. Inherent to their professions, librarians shared a concern about the effect of the World Wide Web on the availability of information to students.
The investigator contacted the managers of the e-mail list serves for both groups and requested permission to send out information about the project and solicitation for voluntary reviewers. Permission was granted provided that the voluntary nature of the project was stated explicitly, reviewers remain anonymous, and clear contact information for the investigator was made available. Volunteer reviewers were given two weeks to review OASIS via the Web. Reviewers could view a Web version of OASIS or download and print a hard copy of the instrument. Directions on how to do both were provided at the Web site. Reviewers then were directed to complete the OASIS Evaluation also via the Web.
Reviews were collected from both groups. Feedback from the KNL was very positive and provided good constructive comments for OASIS. Because reviews were anonymous, however, the investigator did not have the opportunity to ask questions to clarify some comments. In addition to the reviews, several individuals expressed an interest in using the revised OASIS at their schools in the future and contacted the investigator via phone and e-mail to set up possible joint endeavors.
At the time of the one-on-one review, it was not yet determined which classes at Roosevelt High School would be participating in the project. The investigator preferred to work with a student who would not be involved with the project so that he or she did not influence his or her classmates. Therefore, a one-on-one review was conducted with an 11th grade student outside of Roosevelt High School. The student was a self-professed "Internet Junkie" who spent at least two hours a day using the Internet. He approximated that half of his Internet time is spent emailing his friends and that the other half is used to "surf the Web." During times when he had a research project, the student estimated one additional hour per day searching for resources on the Web. He also did not recall ever being introduced to a Web resource evaluation tool in school.
Because he was in the specific age group for whom OASIS was designed, the student was asked to specifically look at the clarity of the directions and questions that accompanied the five sections and the acronyms OASIS itself. The investigator sat with him for about 20 minutes to review OASIS and answer any questions he had, then gave him a week to review it and make written comments. He contacted the investigator the same day to give both written and verbal comments.
The student was able to identify areas where other students would have problems understanding directions to the project. He also pointed out wording that was too complicated for high school students to understand in context. He helped to change the layout of the page so that it was more inviting and less of a worksheet and recommended colored paper for the final instrument. His expediency and comments were extremely useful to and appreciated by the investigator.
The student groups were classes that were taught by the three teachers who volunteered. The three classes were from three different subject areas-science, history, and English. The teachers were instructed to distribute a student demographic survey two weeks prior to visitation by the investigator. The time buffer allowed the investigator time to review the background on each class and plan her presentation accordingly. The investigator met with the students for one class period of 55 minutes in their respective classes. During that time, she first initiated a discussion about the difference between the Internet and World Wide Web, the process of publishing a book versus a Web site, and considerations about how the students looked up information in middle school as opposed to how they find information now. The investigator also asked students to think about what was connected to the Internet. Then, OASIS was introduced to the students and the directions read out loud. Students were given an opportunity to ask questions about the instrument at the end of the period. Students were asked to use OASIS during the next week for a research project and write comments directly on the one-page instrument. Emphasis was made that the instrument was being evaluated and not the students, so the students were encouraged to make comments that would assist in improving the instrument. Both the investigator and teachers also encouraged everyone to use OASIS in any research project and not just limit it to that particular class. After one week, teachers gave the students the OASIS Evaluation. The investigator also returned to the classroom to collect the comments and evaluation and ask students for additional comments.
Only two students wrote comments about OASIS. During the second visit of the investigator, students were hesitant to offer comments about the instrument. Therefore, the investigator was not able to collect specific feedback about the instrument, but only attitude feedback from the OASIS Evaluation form.