Executive Summary

Introduction
This evaluation of the Watershed Action Program (WAP) was conducted to assist the KIDS for the BAY (KftB) staff and program partners with determining the impact and effectiveness of their in-depth watershed program on teacher and student participants, and to a lesser degree students’ families. The evaluation also serves as an invaluable tool in informing future programming and future program evaluation. Forty elementary school teachers and over 1,200 students participated in the WAP Program during the 2006-07 school year. This executive summary focuses on highlights and conclusions of the results from the evaluation process. A full report follows the summary.

The objectives of the Watershed Action Program for students are:

1. Teach young students about their local watershed, how it is connected to the San Francisco Bay, the Pacific Ocean and two National Marine Sanctuaries.

2. Teach students how their own actions affect these water bodies and the organisms that live in them.

3. Provide opportunities for students to connect with a natural watershed habitat by taking them into the field to conduct investigations and explore a creek, bay, or ocean environment.

4. Engage students in hands-on science learning experiences both in their classrooms and in the field—experience they will not otherwise receive.

5. Engage students in service-learning “action projects” designed to teach students how they can become watershed stewards and environmental leaders/teachers in their communities.

6. Improve the health of Alameda and Contra Costa County watersheds, San Francisco Bay, and coastal marine habitats by inspiring students and their families to adopt responsible stewardship behaviors.

The objectives of the Watershed Action Program for teachers are:

1. Provide in-class modeling, training, curriculum resources and support so that they are capable of and confident in including quality environmental science lessons in their classrooms.

2. Provide opportunities for teachers to become comfortable teaching environmental science lessons in the field.

3. Provide an opportunity to earn four to eight units of academic credit through our partnership with California State University East Bay.

To gather the data needed to assess whether these objectives were met, KftB administered a pre- and post-program survey and a written evaluation form to all teacher participants. Twenty-five of the forty teachers completed both surveys, and the results are included in this report. Data from twelve teacher written evaluation forms is also included in this report. Two-hundred students completed pre- and post-program surveys; these students were in classes that were at the fourth grade level and had relatively low numbers of English Language Learners. Results from these surveys are included in this report.

Results: Highlights and Conclusions
Overall, results indicate that the goals of the Watershed Action Program were achieved. The WAP Program provided meaningful watershed experiences for elementary school students, and also provided professional development to classroom teachers in environmental education.

Students showed an overall increase in knowledge of the program content through results from the pre- and post-program surveys. The individual question results reveal that students did learn about their local watershed and its connection to the larger bay and ocean watersheds.

Both the student surveys and the teacher written evaluation forms show student comprehension of how their actions affect the local watershed environment and the animals and plants that share this environment. Many teachers commented that their students are more aware of how their actions can impact the environment. Teachers went on to report that their students have demonstrated a noticeable increase in environmentally-friendly attitudes and behaviors.

Although teachers were forthcoming about how the program has inspired responsible stewardship behaviors in their students, they felt less able to comment on whether the program has influenced students’ families. Some teachers felt that, because they do not regularly interact with their students’ families, they could not properly assess the impact the program has had on students’ home environments.

Many teachers also said that the field trip and the classroom workshops provided their students with new experiences and opportunities to engage in positive hands-on science learning experiences. Teachers went on to say that the program provided experiences their students would otherwise not have been able to participate in. A contributing factor seemed to be that the program content and delivery was appropriate for the age and grade-level of the students.

Teacher survey and written evaluation form results indicate that the program provided professional development opportunities and the resources needed for teachers to feel comfortable teaching environmental science concepts and continuing the WAP Program on their own the following school year. Our direct in-class training model and the program equipment kit proved to be most effective in preparing teachers to teach the program themselves.

Although the majority of teachers felt more comfortable leading an outdoor environmental field trip after participating in the program, the percentage of teachers that felt the same or felt less comfortable after program participation is noteworthy. KftB will investigate further into why teachers may feel less comfortable leading an outdoor field trip after participating in one, so we can improve this component of the program and increase teachers’ confidence levels in this area. Many teachers did express that the outdoor field trip was a valuable and important learning experience, which verifies teachers’ interest and desire in offering these outdoor learning experiences for their students.

Every teacher felt prepared to teach the program to future classes of students after participating in the first year of the program, with most teachers feeling prepared to a considerable or great extent.

Given the results from the gathered evaluation data, KftB concludes that program objectives were largely met and that we should continue to provide the program to elementary school teachers and students. The evaluation process also enlightened us on ways to improve the evaluation tools themselves in order to acquire relevant and useful evaluation data.

Watershed Action Program 2006-07
Evaluation Report

Project Overview

Project Summary
The Watershed Action Program provides 1) meaningful watershed experiences for elementary school students and 2) professional development in the area of environmental education for elementary school teachers. Our specific objectives are as follows:

Student Objectives: Provide meaningful watershed education for 3rd, 4th and 5th grade students from low-income urban elementary schools. The program will:

7. Teach young students about their local watershed, how it is connected to the San Francisco Bay, the Pacific Ocean and two National Marine Sanctuaries.

8. Teach students how their own actions affect these water bodies and the organisms that live in them.

9. Provide opportunities for students to connect with a natural watershed habitat by taking them into the field to conduct investigations and explore a creek, bay, or ocean environment.

10. Engage students in hands-on science learning experiences both in their classrooms and in the field—experience they will not otherwise receive.

11. Engage students in service-learning “action projects” designed to teach students how they can become watershed stewards and environmental leaders/teachers in their communities.

12. Improve the health of Alameda and Contra Costa County watersheds, San Francisco Bay, and coastal marine habitats by inspiring students and their families to adopt responsible stewardship behaviors.

Teacher Objectives: Provide professional development opportunities in the area of environmental education for Alameda County elementary school teachers. The program will:

4. Provide in-class modeling, training, curriculum resources and support so that they are capable of and confident in including quality environmental science lessons in their classrooms.

5. Provide opportunities for teachers to become comfortable teaching environmental science lessons in the field.

6.Provide an opportunity to earn four to eight units of academic credit through our partnership with California State University East Bay.

Program Description
The Watershed Action Program includes five two-hour workshops at the school site and a full day field trip to a local creek, bay or delta habitat. In addition each class selects a watershed action project, which the students plan, develop and implement with guidance from their teacher and support and resources from KIDS for the BAY (KftB).

Teachers learn the programs alongside their students, attend planning and evaluation meetings with KftB instructors and teach preparation and follow up activities from the Curriculum Guide. Each team of teachers receives an equipment kit to continue teaching the program. KftB follows up with teachers in the year after their training to provide additional support and to evaluate success as the teacher teaches the program to her students.

A detailed description of the Watershed Action Program content and activities is located in Appendix A.

Evaluation Goals

KIDS for the BAY’s program evaluation process is both summative and formative. The evaluation data collected from both teacher and student participants during the 2006-07 school year enables us to assess 1) the impact of the program on students and teachers, and 2) the effectiveness of the program content and delivery and how we can improve our teaching methods and curricula for future programming. An added benefit to the evaluation process is being able to assess the effectiveness of the evaluation tools themselves in gathering the desired data from program participants. Information on the types of responses we received, the quality of the responses, and the amount of data we collected from each evaluation tool we administered in 2006-07 will allow us to improve the tools themselves as well as the overall evaluation plan.

Specifically, the Watershed Action Program (WAP) evaluation process seeks to inform us of the following:

Program Effectiveness and Improvement

1. Are we reaching our stated program goals and objectives for teacher and student participants?

2. How can we improve the Watershed Action Program based on the feedback collected from teachers and the results of the evaluation process?

Student Participants

1. What was the students’ overall experience of the program?

2. Was there any increase in student knowledge due to participation in the program?

3. Were there any changes in students’ attitudes, abilities, or behaviors due to participation in the program?

4.What was the impact of the program on students’ families? In particular, have students’ families become more aware of local environmental issues and/or engaged in environmentally responsible behaviors as a result of the program?

Teacher Participants

1. What was the teachers’ overall experience of the program?

2. What suggestions do they have to improve the program content and delivery?

3. Were there any increases in teachers’ comfort level and perceived ability in:

o teaching environmental science concepts
o using the local environment as a learning resource
o facilitating an environmental action project with their students
o leading an outdoor environmental fieldtrip with their students?

4. Do teachers feel prepared to teach the program next year?

5. How useful were the various program components (in-class training, curriculum guide, equipment kit) in providing teachers with what they need to teach the program?

Methods

Quantitative and qualitative evaluation tools were administered to teacher and student participants between September 2006 and June 2007. The Watershed Action Program (intervention) consisted of five in-class workshops, an environmental action project, and a field trip to a creek, bay, or ocean site. Forty 3-5th grade classroom teachers and over 1,200 students participated in the WAP during the 06-07 school year. Some evaluation data was collected from a sample of the entire participant group, while other data was collected specifically from those participants in programs that had a special focus (i.e. National Marine Sanctuaries, steelhead trout). The methodology behind each evaluation tool varies and is described separately for each tool below.

Student Pre- and Post- Program Surveys:
A sample of students completed a pre-survey (Appendix E) before the first classroom workshop, and completed an identical survey within one month of the completion of the classroom workshops. Out of the forty Watershed Action Programs, we selected classes that were at the fourth grade level and that had the lowest number of English Language Learners in order to create a more uniform pool of students. Then we randomly selected classes of students from this pool to complete the surveys.

The educational objectives for each classroom workshop were used as the basis for developing each question on the survey. The surveys were designed to show whether the lesson objectives were met and whether there were any changes in students’ knowledge as a result of participating in the WAP. The surveys contained a variety of different question-and-answer styles and techniques (i.e. multiple choice, fill-in-the-blank, pictograms) that are appropriate and suitable for the age of the student participants (9-11 year olds). A variety of questioning strategies were used. Some questions simply checked for knowledge while others required critical thinking strategies and/or more depth of knowledge.

Two-hundred students completed the pre- and post-program surveys. Each classroom teacher administered the surveys using a script (Appendix I and J) to introduce and facilitate the survey process. Students used identification numbers instead of their names to identify themselves on their surveys. The anonymity of the surveys helped students to understand that this was not a “test” that would be included in their school grades.
Within our smaller pool of student participants (fourth grade level and low number of E.S.L. students), we also selected classes who were participating in programs that had a special focus to complete additional questions that checked for knowledge around specific information. Seventy students taking part in the Marine Sanctuaries WAP completed one additional survey question (Appendix F) that focused on the marine sanctuaries. Sixteen students who participated in the WAP in the city of Richmond completed one additional question (Appendix G) about how to lesson their exposure to mercury. Fifty-two students taking part in the Steelhead Trout WAP completed two additional questions (Appendix H) that focused on the trout’s migration to/from creek and ocean habitats during its different life stages, and creek habitat needs of local steelhead trout.

Teacher Pre- and Post-Program Surveys:
Each teacher participant (40 total) completed a short pre-program survey (Appendix B) in September-October 2006, before the program started. They then completed a longer post-program survey (Appendix D) that contained the same questions as the pre-program survey, plus additional questions. This survey was administered at the completion of all program activities (including the field trip and action project). The pre-post survey questions contained standard likert-scale response options and asked teachers their comfort level in various areas such as leading an environmental field trip with their students and teaching environmental science concepts. The additional questions included in the post-survey asked how the resources we provided and the program structure enabled them to teach the program themselves. Results for the pre- and post-program surveys reflect the data collected from any of the 40 teacher participants who completed and turned in both surveys. Twenty-five teachers completed both surveys, and two additional teachers completed just the post-survey with additional questions.

Teacher Written Evaluation Forms:
Each teacher participant completed written evaluation forms (Appendix C and D) that asked open-ended questions about their overall impressions of the program, program highlights, suggestions for program improvement, and how the program has impacted them, their students, and the school community. Teachers completed one written evaluation form at the completion of the classroom workshops, and completed another at the close of all program activities. The open-ended questions allowed teachers to be insightful in their responses and also allowed them to be detailed in their answers. Results from twelve WAP teacher participants are included in this report.

Data Analysis

Student Pre- and Post Program Surveys:
Each question on the survey was given a specific point score (weight) and the total point score for the survey equaled 100. Some questions were given a higher weight than others based on what we felt was important for the students to know and understand. In other words, we prioritized the concepts and information we wanted students to know and assigned weight to each survey question accordingly. If we felt if was important for the students to know a certain concept, the correlating question was given a higher weight than other questions on the survey.

Each class who completed the surveys was given a class code to identify them. We then compiled the pre-program and post-program surveys for each student using their identification numbers and discarded any surveys that did not have both a pre- and a post. Each pair of surveys was then given a new identification code. This code was recorded on both the surveys and the excel spreadsheet. The pre-surveys were graded and the results for each question were recorded in an excel spreadsheet. We also calculated the total point score for each survey and put this in a separate column. The post-surveys were graded and the results were entered into a separate excel spreadsheet.

Using the software program XLSTAT, we compared the pre-survey results to the post-survey results using a paired t-test. The survey results were compared to see if there was a significant increase in students’ knowledge due to participation in the Watershed Action Program.

Teacher Pre- and Post-Program Surveys:
The pre-survey and post-survey responses from each teacher were put into a table and compared to see what, if any, changes occurred as a result of the intervention. The changes in responses were put into a separate table and then compiled into three percentage statements (i.e. increase, no change, or decrease) for each survey question. These are shown in the Results section. Results are shown for all WAP teacher participants that completed and turned in both pre- and post-program surveys. Twenty-five teachers completed both surveys, and two additional teachers completed just the post-survey with additional questions. The responses to the questions that were asked only in the post-program survey were compiled and displayed in a separate table.

Teacher Written Evaluation Forms:
The qualitative data we received from teachers’ written evaluation forms was coded and quantified. We reviewed the fifteen questions posed on the evaluation forms and selected a few key questions to formally analyze. We then read and re-read all of the responses to each of the selected questions, eventually drawing out common themes and categories that emerged from the responses. As the responses were repeatedly reviewed, we merged some categories and created sub-categories where needed. Once we completed the list of categories, we assigned code words to identify each category. The responses to each question were then coded and put into a table. As responses were put into the table, we took note of how often each type of response was made. The results for the written evaluation forms are displayed using this information.

Results

Student Pre- and Post-Program Survey Results

Whole Test Results

Results Summary: Paired t-test results from 200 fourth grade students determined that there was a statistically significant increase in knowledge after experiencing the Watershed Action Program intervention (t199 =19.51, p<0.0001). The total possible score for the entire test was 100 and the mean score increase between pre- and post-tests was 23.52 points.

Individual Question Results

Questions 1 and 2 asked students to name their local creek watershed and identify things that are a part of their watershed.
Results Summary: Results show a significant increase in knowledge about watersheds.

Questions 5 and 11 checked students’ knowledge about the storm drain system. Question 5 checked to see if students knew that storm drains connect to a local body of water. Question 11 asked students to identify, from a list of general actions, which actions could cause storm drain pollution.
Results Summary: Results show a significant increase in knowledge about the storm drain system and potential pollutants.

Question 10 checked to see if students knew how pollution in the San Francisco Bay can harm human health through catching and eating fish from the bay.
Results Summary: Results show a significant increase in knowledge about how Bay pollution can harm humans through the food chain.

Questions 14 through 17 asked students to identify and label various locations on a map of the San Francisco Bay Watershed.
Results Summary: Results show a significant increase in knowledge of San Francisco Bay geography.

Additional Question Results

National Marine Sanctuaries WAP: The students who learned about the National Marine Sanctuaries were asked to identify the names of the local marine sanctuaries from a list. Seventy students completed this additional question.
Results Summary: Results show a statistically

City of Richmond WAP: Students in the City of Richmond learned about ways to reduce their exposure to mercury from eating seafood. These students were asked to identify ways that they and their families can reduce their exposure to mercury from a list of actions.
Results Summary: Results show a statistically significant increase in knowledge (t15 = 1.379, p<0.094) about reducing mercury exposure from eating seafood. The total possible score was 100.

Steelhead Trout WAP: The students who learned about salmonids, and in particular steelhead trout, were asked two additional questions. One question focused on where steelhead trout live (creek or ocean) during the different stages of their lives; the other question asked students to identify various creek features that steelhead trout need in order to survive.
Results Summary: Results show a statistically significant increase in knowledge for both questions (ocean/creek life stages: t51 = 2.336, p<0.012) (creek habitat needs: t51 = 2.336, p<0.012). Each question was worth 100 points total.

Teacher Pre- and Post-Program Survey Results

Question 1: To what extent do you feel comfortable using the local watershed environment as a learning resource? N = 25
Results Summary: The majority of teachers felt more comfortable using the local watershed environment as a learning resource after participating in the WAP Program. A considerable amount of teachers did not change their level of comfort in using the local watershed environment as a learning resource.

8% of teachers felt less comfortable after participating in the WAP Program.
36% felt the same, no change, after participating in the WAP Program.
56% teachers felt more comfortable after participating in the WAP Program.

Question 2: To what extent do you feel comfortable teaching environmental science concepts? N = 25
Results Summary: The majority of teachers felt more comfortable teaching environmental science concepts after participating in the WAP Program. A considerable amount of teachers did not change their level of comfort teaching environmental science concepts.

16% of teachers felt less comfortable after participating in the WAP Program.
32% felt the same, no change, after participating in the WAP Program.
52% teachers felt more comfortable after participating in the WAP Program.

Question 3: To what extent do you feel comfortable leading an outdoor environmental field trip with your class? N = 25
Results Summary: The majority of teachers felt more comfortable leading an outdoor environmental field trip with their class after participating in the WAP Program. A considerable amount of teachers did not change their level of comfort in leading an outdoor environmental field trip, and a notable amount of teachers felt less comfortable leading an outdoor environmental field trip after participating in the WAP Program .

24% of teachers felt less comfortable after participating in the WAP Program.
36% felt the same, no change, after participating in the WAP Program.
40% teachers felt more comfortable after participating in the WAP Program.

Question 4: To what extent do you feel comfortable facilitating an environmental action project with your class? N = 25
Results Summary: The majority of teachers felt more comfortable facilitating an environmental action project with their class after participating in the WAP Program. A considerable amount of teachers felt less comfortable facilitating an environmental action project.

40% of teachers felt less comfortable after participating in the WAP Program.
16% felt the same, no change, after participating in the WAP Program.
44% teachers felt more comfortable after participating in the WAP Program.

Questions 5 – 8: Post-Program Questions
Results Summary: All classroom teachers feel prepared to teach the WAP Program themselves after participating in the program, with the majority of teachers feeling prepared ‘to a considerable extent’. All teachers felt that the resources we provided them enabled them to teach the program themselves, with the ‘in-class modeling’ and the ‘equipment kit’ having the highest ratings. A few teachers did not respond to question 8 because they had not received the equipment kit at that time and therefore did not know how to respond.

Teacher Written Evaluation Form Results

Question 1: Please give your overall impressions of the classroom workshops.

N = 11

Results Summary: The majority of teachers expressed overall satisfaction, enjoyment, and appreciation of the WAP Program. Many teachers commented on how the program had a beneficial impact on their students. Many teachers also wrote positive comments about the classroom workshop content. There were no negative comments in this section.

Good Workshop Content (Content)

Satisfaction with KftB Instructors (Instructors)

Overall Appreciation (Overall)

Impact on Students (StudImpact)

Good Curriculum (Curriculum)

Question 2: Please describe one or two highlights from the classroom workshops.

N = 11

Results Summary: By far the majority of teachers mentioned the hands-on activities as the highlight of the classroom workshops. A few hands-on activities were cited specifically, with the San Francisco Bay-Delta Watershed Model receiving the highest number of comments. Many teachers also mentioned that the Neighborhood Pollution Survey and Clean-up was a highlight.

Hands-on Activities (Hands-on)

Good Visual Aids and Hand-Outs (Materials)

Continual Reference to Concepts to Reinforce Learning (Reference)

Connection Between Pollution and Effects on Wildlife (Connection)

Strong Vocabulary Building (Vocab)

Overall Enjoyment (Enjoy)

KftB Supplied Everything Needed for Each Lesson (Supplies)

Relevance to Students’ Communities/Experiences (Relevance)

Provided New Opportunities for Students (Opportunities)

Question 3: Please suggest any improvements to the Watershed Action Program classroom workshop component. N = 11

Results Summary: Five of the eleven teachers had no suggestions to improve the classroom workshops of the Watershed Action Program. The majority of the remaining comments focused on supplementing the program with additional classroom workshops, more hands-on activities, and language arts activities to reinforce the hands-on activities.

No Improvement Needed (NoImprove)

Include Supplemental Activities (Supplemental)

Too Much Repetition of Concepts (Repetition)

Break Up/Limit Lectures (Lectures)

EJ Piece Not Connected to Program Purpose (EJ)

Question 4: Please describe the overall experience of the field trip for you and your students. N = 7

Results Summary: All responses about the overall experience of the field trip were positive. Many teachers expressed that their students enjoyed the experience; a few teachers said that the field trip took their students to a place they had never been before. A few teachers also cited specific activities as highlights.