2/14/02 Brownbag Presentation
KCC Curriculum Evaluation Project
Over the past two semesters I have conducted participatory action research into the implementation of the Generation 1.5 ESL Curriculum at Kapiolani Community College. In the following presentation, I will discuss my preliminary research findings based on a participatory action research approach to evaluation. I will first briefly talk about curriculum evaluation in general before presenting my particular approach along with the model that I have developed for curriculum evaluation. Next I will highlight a number of findings that suggest success with the curriculum implementation. Finally, I will go over some of the difficulties encountered throughout the curriculum implementation and the solutions that have been proposed to counter these difficulties.
Evaluation is a necessary requirement of any curriculum. It has been called by Brown (1995) “the glue that connects and holds all of the (curricular) elements together.” Without evaluation, a curriculum lacks meaning, and therefore, lacks credibility.
Evaluators may choose different approaches and different methodologies when embarking on their research projects depending on the intended goals of their evaluations. An evaluator may take the outsider approach due to time constraints, when the etic perspective of the researcher is valued, or when no other option is available. On the other hand, an evaluator may seek the insider perspective when time constraints are not a factor and when the emic viewpoints of the project’s participants are of primary concern.
The evaluator with time constraints may succumb to the “jet-in/jet-out” approach to evaluation, a situation where the researcher enters a research site, quickly gathers data, and then exits the site as quickly as possible, ideally with little disturbance to the project’s participants. This approach is most likely conducted with quantitative methods. Conversely, a project may require a longitudinal study in order to conduct more thorough research. Longitudinal research studies may utilize quantitative methods, qualitative methods, or a combination of the two methodologies.
Evaluations generally take on two types: summative or formative. Summative evaluations seek to determine a curriculum’s success and usually result in drastic changes. A project’s requirements may call for a summative evaluation when a program has reached its endpoint. However, summative evaluations may also be carried out in the middle of a curriculum project to allow for analyses of activities up to that point. Formative evaluations, on the other hand, seek to gather and analyze data that contributes to the improvement of the curriculum. These types of evaluations usually resulting in minor adjustments and changes to the curriculum.
In the curriculum project of which I am currently engaged, I am taking a participatory action research approach to evaluation. I participate by means of co-teaching some times, tutoring some times, and joining student discussion groups at other times. As a participating member of the classroom, I predominantly take the insider’s perspective. Therefore, I am in an advantageous position to seek the emic perspective of the other participants in the class. This evaluation project is a longitudinal study, with projected data gathering over a one-year period. Also drawing from Brown (1995), this study meets the desired requirements that a curriculum evaluation is ongoing and continuous throughout the life of the program. In addition, the goals of this evaluation project are both formative and summative, yet without any desired or intended outcomes. Lastly, this study relies on both qualitative as well as quantitative methods in its data collection and analysis.
I have developed the KCC Participatory Curriculum Evaluation Model, or PaCE, to graphically reflect the comprehensive nature of the actual evaluation project. It is intended to operate in conjunction with the curriculum continuously throughout its implementation, from beginning until end. Therefore, at the center of the PCE model lays the ARC curriculum as the focus of the evaluation. The model includes all of the participants involved in the curriculum implementation project and the types of data that each one contributes to the evaluation. The structure of the model is devised to show the relationship between all of the participants involved. Hence, coordination between and among all of the participants involved with the curriculum implementation is a requirement of the model.
Having briefly discussed my evaluation approach for the current research project and the model that I have developed to guide my evaluation, I would like to present some of my preliminary findings, the first of which point to successes with the curriculum implementation.
Perhaps one of the more promising findings thus far is that the Generation 1.5 ESL Curriculum appears to be applicable to a much broader audience than just to the students for which it was developed. Originally intended for implementation in a class composed predominantly of generation 1.5 students, the first semester of the research project found the curriculum implemented in a very different classroom. Of the 19 students still enrolled in the course at the end of the fall semester of 2001, only five were generation 1.5 students, all of whom had graduated high school within the previous year. The remaining 14 students were either exchange students or recent immigrants. Of these 14 students, nine had graduated from high school only, and the remaining five had received varying degrees of college education prior to enrolling in KCC. In addition, the 19 students enrolled in the course came from a variety of countries and had different first language backgrounds. This student diversity coupled with the end of the semester assessment figures indicates that the Generation 1.5 ESL Curriculum may be applicable to more college ESL contexts than for which it was originally intended. A suggestion for future research would be to test this hypothesis in different ESL contexts with different student populations, for instance, in a linked ELI 70 and 73 course in this department.
Another indicator of the curriculum’s success, which also supports the findings previously mentioned, is the completion rate and quality of the students’ ethnographic research projects. Of the 19 students enrolled in the course at the end of the semester, 16 students submitted complete reports. All of these reports were assessed by the teacher as adequate enough to receive a grade of “Pass” for the course, equivalent to at least the 80% minimum required to receive such a grade. Therefore, the students’ overall passing rate in this course was 84%, considered above average for this department. Further data analysis will be conducted to compare this course’s passing rate to the passing rates of other sections of ESOL 94 before this initial finding can be corroborated. However, the preliminary results seem promising.
Along with the numerous successes observed throughout this curriculum implementation project, we have also experienced quite a few difficulties. One difficulty evidenced has been the students’ overall understandings of the curriculum goals. During individual and group interviews over the mid-term period last semester, I sought to find out what students believed were the primary objectives of the course. While the students overwhelmingly expressed positive attitudes about the course in general, many students expressed that they were unclear about the purposes of some of the course lessons and the overall goals of the course itself. Eric, a generation 1.5 student, expanded on the common response of, “I don’t know,” by saying
I’m not really sure why we’re learning this kind stuffs, but I do know it’s different. I never learn this things in high school. I’m sure it’s gonna be good for my future, but I really need to improve my grammar.
In a group interview with several students from the course, Diana, a first-semester exchange student, provided the following insight:
My friend is took the other (ESOL 94) class, and she told me the instructor lecture like this, so when I heard that… I thought, oh, my class is a little bit different from her class…because you know the other class is focus on the textbook. They take a test or they are reading the textbook or something. But my class, just one time we study the textbook or something, so even though we bought the textbook but we didn’t have any chance to use the textbook… we just use some papers and turn in or something… So, I was so confused, why she want to teach this one… why she don’t use the textbook? But…later I got understand, oh, what is her purpose in the class…she just want to teach us how to write the research paper. And then I understand her class activity.
These comments are not surprising considering the conditioning that these students have received throughout their schooling. Due to his high school experience, Eric expected the course to emphasize grammar instead of its focus on actual language use and production. Although he is able to speculate about his future, immediate benefits may be obscured by his preoccupation with grammar.
In Diana’s case, her initial confusion stemmed from the absence of textbook use in her experimental class as compared to her friend’s experience in a standard section of the same course. She has been conditioned that teachers and students should follow the textbook ordering of lessons and activities instead of an innovative lesson plan. Part of this situation is due to the department’s instruction that teachers make a textbook a requirement of the course. Teachers are also told that they should use certain sections of the textbooks for their particular courses. However, the section of the textbook designated for this course was found by the teacher to be incongruent with the curricular objectives; therefore, it was used infrequently over the semester. This situation has lead to students questioning the necessity of the textbook and the overall goals of the course when the textbook is not being used.
Additionally, Diana reached the conclusion that the overall purpose of the course is to learn how to write a research paper. This goal is in fact just the end result of a curriculum designed to improve students’ discursive and linguistic proficiency and help students develop an improved understanding of the social and educational expectations of teachers at U.S. colleges. Diana’s conclusion indicates that at the time of the interview, she was still unclear about some of the activity purposes and course goals.
During a meeting at the end of the semester to discuss preliminary research findings, the teacher and I discussed this notion of explicit awareness of goals. In response to my suggestion that every aspect of the course should be made as explicit as possible, she responded
Yes, I think that is true, but at the same time, I think that understanding some of this stuff is a process, and that even if I’m explicit about what I’m trying to accomplish they may not really understand it till toward the end.
This highlights some of the challenges associated with implementing an innovative curriculum such as this. Since an integral part of this curriculum is to develop meta-cognitive and academic awareness, the students must go through a process of exposure, practice, and discovery before ideas become their own and they truly understand the nature of the complex task in which they are apprenticed.
Another relevant concern that emerged during student interviews was the feeling that the course did not have enough different kinds of academic writing assignments over the semester. Several students seemed to focus on the final research report as the exclusive form of academic writing required over the semester. Cindy, another exchange student and two-year college graduate in her home country, summed up these students’ opinions when she said
I already learned about library and doing research on Internet. I want to know more about essays… I think I’m gonna need to know how to write different kinds of essays in my other college class.
It appears that Cindy did not consider some of the regular class assignments such as journals, free-writes, summaries, outlines, discussions, and responses to be as valuable to her future academic career as essays could be. Perhaps she has also been conditioned by her past schooling into believing the traditional notion that mastery of the 5-paragraph essay will equal academic success. On the other hand, it is also possible that the curriculum should include more diverse academic writing assignments to address students’ future academic needs while at the same time scaffolding the final research project.
For each of the difficulties previously mentioned, solutions have already been proposed and implemented this semester in an effort to improve the curriculum. To address the issue of students’ understanding of the goals of the curriculum, the teacher is making more of an effort to describe the relationships between individual lessons and curriculum objectives. In addition to a thorough introduction to the course and its overall goals at the beginning of this semester, the teacher is explicitly stating how each new activity links to previous activities and to the course goals. It is hoped that by consistently showing these relations, students will have a much better understanding of how the entire course will contribute to their academic careers. Thus far, initial data seems positive; however, much more is needed to determine whether or not this strategy yields successful results.
Regarding textbook requirements, the teacher has also negotiated the curriculum to include more use of the textbook. She has decided to use a different section of the textbook that focuses on language and culture for her class. These topics are much more consistent with the lessons, activities, and overall goals of the course. Hopefully, this move will solve the problem of student confusion regarding textbook use.
To counter the perceived shortage of different kinds of academic writing assignments over the semester, the instructor has negotiated the curriculum to provide for a greater variety of writing assignments aimed at scaffolding the final ethnographic research project. Her first step has been to reorganize and focus individual writing lessons so that each assignment more logically builds upon the previous one from the beginning of the course to the final research report. The teacher has also included more types of academic writing activities in her lessons this semester, both formal and informal, including analyses, interpretations, and narrative essays. Preliminary findings suggest that this solution may alleviate future student concerns about the quantity of academic writing assignments in the course, but more data is needed from the students’ perspectives before any firm conclusion to this effect can be reached.
Although this research report presents tentative findings, the results seem very promising for the future development of the Generation 1.5 ESL Curriculum. Due to the formative and participatory nature of the curriculum evaluation component, successes and difficulties will continue to emerge with recommendations for improvement as the curriculum implementation project evolves. It is hoped that this evaluation project will provide information on curriculum and pedagogical practices that foster improved educational practices for generation 1.5 college students in Hawaii and nation-wide.
contents (c) 2002 Shawn
Ford/ Webb-Ed Press