KIDS for the BAY

___________________________________________________

Storm Drain Rangers Program 2006-07

Evaluation Report

Executive Summary

Introduction
This evaluation of the Storm Drain Rangers Program (SDR Program) was conducted to assist the KIDS for the BAY (KftB) staff and program partners with determining the impact and effectiveness of their in-depth storm drain pollution reduction program on teacher and student participants, and to a lesser degree students’ families. The evaluation also serves as an invaluable tool in informing future programming and future program evaluation. Twenty elementary school teachers and over 600 students participated in the SDR Program during the 2006-07 school year. This executive summary focuses on highlights and conclusions of the results from the evaluation process. A full report follows the summary.

The goals of the Storm Drain Rangers Program are:

Specific Storm Drain Rangers lesson objectives include:

Lesson One

Lesson Two

Lesson Three

To gather the data needed to assess whether these objectives were met, KftB administered a pre- and post-program survey and a written evaluation form to all teacher participants. Nine of the twenty teachers completed both surveys, and the results are included in this report. Eight of the twenty teachers completed written evaluation forms; feedback from these evaluation forms are included in this report. Twenty fourth grade students completed a pilot pre- and post-program survey. Results from these surveys are included in this report.

Results: Highlights and Conclusions

Overall, evaluation results indicate that the goals of the Storm Drain Rangers Program were achieved. The SDR Program provided meaningful hands-on watershed and storm drain education for elementary school students, and also provided professional development to classroom teachers in environmental education.

Students showed an overall increase in knowledge of the program content through results from the pre- and post-program surveys. The individual question results reveal that students did learn about their local watershed and its connection to the larger bay and ocean watersheds. Students also increased their awareness of the storm drain system and its connection to local waterways (i.e. creeks), the bay, and the ocean. An increase in knowledge about potential storm drain pollutants was also evident through the pre- and post-program surveys.

Both the student surveys and the teacher written evaluation forms show student comprehension of how their actions affect the local watershed environment and the animals and plants that share this environment. Students increased their awareness about the storm drain system and potential pollutants that can enter the system and affect the local creek, the bay, and the ocean. Many teachers commented that their students are more aware of how their actions can impact the environment. Teachers went on to report that their students have demonstrated a noticeable increase in environmentally-friendly behaviors, especially in the area of litter prevention (i.e. not littering anymore, telling others not to litter). A few teachers stated that their students’ attitudes about pollution and the environment have changed. These observations of positive changes in students’ awareness, attitudes, and behaviors are coming from teachers who are in close and constant contact with their students. This type of information about students is more valid coming from teachers than from students self-reporting about any changes they have made due to participation in the program. This is because the teachers serve as an ‘outside’ objective observer, whereas a variety of factors can come into play and lessen the validity of the observations if it comes from the students themselves.

Teacher survey and written evaluation form results indicate that the program provided professional development opportunities and the resources needed for teachers to feel comfortable teaching the SDR Program on their own the following school year. Our direct in-class training model, access to program equipment kit, and the comprehensive curriculum guide proved to be effective in preparing teachers to teach the program themselves. Every teacher responded that they plan to teach the SDR Program in their classrooms in the future.

The results to some survey questions also show how the program was a positive experience for teachers. Every teacher responded that they would recommend the program to other classroom teachers, which is an indication that they felt the program was valuable and worthwhile. All teachers also said that the program met their expectations. The program also inspired teacher participants to incorporate more hands-on science activities in their school-year curriculum.

Given the results from the gathered evaluation data, KftB concludes that program objectives were largely met and that we should continue to provide the program to elementary school teachers and students. The evaluation process also enlightened us on ways to improve the evaluation tools themselves in order to acquire relevant and useful evaluation data.

Storm Drain Rangers Program 2006-07

Evaluation Report

Program Overview

Program Summary

The Storm Drain Rangers Program provides meaningful hands-on watershed and storm drain education for elementary school students and professional development in the area of environmental education for elementary school teachers. Specific program objectives are as follows:

The goals of the Storm Drain Rangers Program are:

Specific Storm Drain Rangers lesson objectives include:

Lesson One

Lesson Two

Lesson Three

Program Description

The Storm Drain Rangers Program includes two 2.5-hour workshops and one 45-minute final lesson that teaches students about the storm drain system and its connection to the local creek, San Francisco Bay, and Pacific Ocean, and about ways to reduce pollution to the storm drain system and consequently our waterways.

Teachers learn the programs alongside their students, attend planning and evaluation meetings with KftB instructors and teach preparation and follow up activities from the Curriculum Guide. Each team of teachers has access to program equipment through our loaning system to continue teaching the program every school year to each new class of students. KftB follows up with teachers in the year after their training to provide additional support and to evaluate success as the teacher teaches the program to her students.

A detailed description of the Storm Drain Rangers Program content and activities is located in Appendix A.

Evaluation Goals

KIDS for the BAY’s program evaluation process is both summative and formative. The evaluation data collected from both teacher and student participants during the 2006-07 school year enables us to assess 1) the impact of the program on students and teachers, and 2) the effectiveness of the program content and delivery and how we can improve our teaching methods and curricula for future programming. An added benefit to the evaluation process is being able to assess the effectiveness of the evaluation tools themselves in gathering the desired data from program participants. Information on the types of responses we received, the quality of the responses, and the amount of data we collected from each evaluation tool we administered in 2006-07 will allow us to improve the tools themselves as well as the overall evaluation plan.

Specifically, the Storm Drain Rangers Program evaluation process seeks to inform us of the following:

Program Effectiveness and Improvement

Student Participants

Teacher Participants

Methods

Quantitative and qualitative evaluation tools were administered to teacher and student participants between September 2006 and June 2007. The Storm Drain Rangers Program (intervention) consisted of three in-class workshops totaling 5.75 hours of direct instruction. Twenty 3-5^th grade classroom teachers and over 600 students participated in the SDR Program during the 06-07 school year. Some evaluation data was collected from a sample of the entire participant group, while other data was collected from the entire participant group. The methodology behind each evaluation tool varies and is described separately for each tool below.

Student Pre- and Post- Program Surveys:

A sample of students completed a pre-survey (Appendix B) before the first classroom workshop, and completed an identical survey within one month of the completion of the program. Because the SDR Program and this student evaluation tool was new, we decided to pilot this instrument by choosing one fourth grade class to complete the surveys. Results from the pilot of this survey will be used to make adjustments to the survey questions in order to improve students’ comprehension of the questions and improve the types of data we collect from students through this survey.

The educational objectives for each classroom workshop were used as the basis for developing each question on the survey. The surveys were designed to show whether the lesson objectives were met and whether there were any changes in students’ knowledge as a result of participating in the SDR Program. The surveys contained a variety of different question-and-answer styles and techniques (i.e. multiple choice, fill-in-the-blank, pictograms) that are appropriate and suitable for the age of the student participants (9-11 year olds). A variety of questioning strategies were used. Some questions simply checked for knowledge while others required critical thinking strategies and/or more depth of knowledge.

Twenty students completed the pre- and post-program surveys. The classroom teacher administered the surveys using a script (Appendix C and D) to introduce and facilitate the survey process. Students used identification numbers instead of their names to identify themselves on their surveys. The anonymity of the surveys helped students to understand that this was not a “test” that would be included in their school grades.

Teacher Pre- and Post-Program Surveys:

Each teacher participant (20 total) completed a short pre-program survey (Appendix E) in September-October 2006, before the program started. They then completed a longer post-program survey (Appendix G) that contained the same questions as the pre-program survey, plus additional questions. This survey was administered at the completion of all program activities. The pre-post survey questions contained standard likert-scale response options and asked teachers their comfort level in various areas such as teaching environmental science concepts. The additional questions included in the post-survey asked how the resources we provided and the program structure enable them to teach the program themselves. Results for the pre- and post-program surveys reflect the data collected from any of the 20 teacher participants who completed and turned in both surveys (9 teachers total).

Teacher Written Evaluation Forms:

Each teacher participant was asked to complete a written evaluation form (Appendix F) that asked open-ended questions about their overall impressions of the program, program highlights, suggestions for program improvement, and how the program has impacted them and their students. The open-ended questions allowed teachers to be insightful in their responses and also allowed them to be detailed in their answers. Eight teachers completed and turned in the evaluation form, and results are included in this report.

Data Analysis

Student Pre- and Post Program Surveys:

Each question on the survey was given a specific point score (weight) and the total point score for the survey equaled 100. Some questions were given a higher weight than others based on what we felt was important for the students to know and understand. In other words, we prioritized the concepts and information we wanted students to know and assigned weight to each survey question accordingly. If we felt if was important for the students to know a certain concept, the correlating question was given a higher weight than other questions on the survey.

We compiled the pre-program and post-program surveys from each student using their anonymous identification numbers and discarded any surveys that did not have both a pre- and a post. Each pair of surveys was then given a new identification code. This code was recorded on both the surveys and the excel spreadsheet. The pre- and post-surveys were graded and the results for each question were recorded in separate excel spreadsheets. We also calculated the total point score for each survey and put this in a separate column.

Using the software program XLSTAT, we compared the pre-survey results to the post-survey results using a paired t-test. The survey results were compared to see if there was a statistically significant increase in students’ knowledge due to participation in the Storm Drain Rangers Program.

Teacher Pre- and Post-Program Surveys:

The pre-survey and post-survey responses from each teacher were put into a table and compared to see what, if any, changes occurred as a result of the intervention. The changes in responses were put into a separate table and then compiled into three percentage statements (i.e. increase, no change, or decrease) for each survey question. These are shown in the Results section. Results are shown for all SDR Program teacher participants that completed and turned in both pre- and post-program surveys. Nine teachers completed both surveys, and nine teachers completed just the post-survey with additional questions. The responses to the questions that were asked only in the post-program survey were compiled and displayed in a separate table.

Teacher Written Evaluation Forms:

We reviewed the four questions posed on the evaluation forms and chose three key questions to formally analyze. The qualitative data we received from teachers’ written evaluation forms was then categorized and quantified. We read and re-read all of the responses to each of the selected questions, eventually drawing out common themes and categories that emerged from the responses. As the responses were repeatedly reviewed, we merged some categories and created sub-categories where needed. As responses were placed into the final categories, we took note of how often each type of response was made. The results for the written evaluation forms are displayed using this information. The response category is listed first, and then any specific statements that were made that related to the category are listed below. The number in parentheses next to the category/statement indicates the number of responses made for that category/statement. Eight teachers completed the written evaluation form; the results for each selected question from these eight teachers are included in this report.

Results

Student Pre- and Post-Program Survey Results

Whole Test Results

Results Summary: Paired t-test results from 20 fourth grade students determined that there was a statistically significant increase in knowledge after experiencing the Storm Drain Rangers Program intervention (t₁₈ =2.369, p<0.015). The total possible score for the entire test was 100 and the mean score increase between pre- and post-tests was 10.79 points.

Individual Question Results

Questions 1 and 2 asked students to name their local creek watershed and identify things that are a part of their watershed.

Results Summary: Results show a statistically significant increase in knowledge about watersheds.

Questions 5 and 7 checked students’ knowledge about the storm drain system. Question 5 checked to see if students knew that storm drains connect to a local body of water. Question 7 asked students to identify, from a list of general actions that cause pollution, which actions could cause storm drain pollution.

Results Summary: Results show a significant increase in knowledge about the storm drain system and potential storm drain pollutants.

Questions 3 and 4 asked students to identify the two types of water (salt and fresh) found in the San Francisco Bay estuary, and the bodies of water that bring these two types of water into the Bay.

Results Summary: Results show a statistically significant increase in knowledge about the types of water found in the bay and where the fresh and salt water originate from before they enter the Bay.

Question 8 checked for students’ knowledge about, out of all the water in the world, how much of it is fresh water.

Results Summary: Students showed a statistically significant increase in knowledge about the amount of fresh water in the world.

Question 9 asked students to identify, from four pictures of pollution, which pictures were examples of point source pollution.

Results Summary: Students showed a statistically significant increase in knowledge about the types of point source pollution.

Teacher Pre- and Post-Program Survey Results

Teachers were asked to respond to the following statements using the response options: ‘strongly disagree’, ‘disagree’, ‘neutral’, ‘agree’, and ‘strongly agree’. Any changes in teachers’ responses from pre to post test were noted and recorded in the following percentage statements.

Statement: Storm drain pollution is a problem in the school’s neighborhood. N = 9
Results Summary: Results show that most teachers increased their concern about storm drain pollution in their school’s neighborhood.

11% of teachers decreased their concern after participating in the Storm Drain Rangers Program.

33% of teachers had the same level of concern after participating in the Storm Drain Rangers Program.

56% of teachers increased their concern after participating in the Storm Drain Rangers Program.

Statement: I feel confident using the local watershed environment as a learning resource. N = 9

Results Summary: All teachers either did not change their level of confidence, or increased their level of confidence, in using the local watershed environment as a learning resource.

0% of teachers decreased their confidence level after participating in the Storm Drain Rangers Program.

56% of teachers felt the same, no change, after participating in the Storm Drain Rangers Program.

44% increased their confidence level after participating in the Storm Drain Rangers Program.

Statement: I feel confident teaching environmental science concepts. N = 9

Results Summary: The majority of teachers maintained the same level of confidence in teaching environmental science concepts. Some teachers did increase their confidence in this area.

11% of teachers decreased their confidence level after participating in the Storm Drain Rangers Program.

56% of teachers felt the same, no change, after participating in the Storm Drain Rangers Program.

33% increased their confidence level after participating in the Storm Drain Rangers Program.

Teacher Post-Program Survey Results

Teachers were asked to complete the following questions after they had completed the SDR Program. The questions asked for feedback on a variety of matters, including: how the resources we provide to them enable them to teach the program themselves; whether they would recommend this program to other classroom teachers; whether they plan to incorporate more hands-on science in their classrooms; whether the program has increased their students’ enthusiasm for science; whether the program met their expections; and whether they plan on teaching the program themselves in their classrooms.

Results Summary: Results are positive in all areas, with teachers wanting to incorporate more hands-on science in their classrooms and the program increasing their students’ enthusiasm for science rating the highest.

Teacher Written Evaluation Form Results

Question: Please share any highlights from the classroom workshops. N = 8

Results Summary: Five teachers listed the storm drain/urban run-off pollution survey and neighborhood clean-up as a highlight of the program. Five teachers also mentioned the hands-on activity of building a clay model of the San Francisco Bay watershed as a highlight. A few teachers appreciated the hands-on nature of the classroom workshops. The remaining comments included a variety of different highlights to the program. All responses were positive.

- Pollution/storm drain survey and neighborhood clean-up (5)

- Students loved building the Bay model (5)

- Hands-on activities were meaningful (3)

- Exposure to the bay and watersheds (1)

- Students learned and used new vocabulary (1)

- Enthusiastic KftB Instructor (1)

- Map reading skills (1)

- Extensions to other lessons/activities outside of program (1)

- Excellent visuals (1)

- Great way to introduce environmental stewardship (1)

Question: Please share the impact that the lessons and activities have had on your students (if any). Have you noticed your students changing their behavior as a result of the program? N = 8

Results Summary: Teachers cited many ways that students have increased their awareness and knowledge due to the SDR Program. Many teachers also gave examples of how students have positively changed their behaviors towards the environment as a result of participating in the program. Two teachers referenced how their students’ attitudes have changed in a positive way. All but one comment showed how the program has positively impacted student participants; one teacher felt that more time was needed to assess real behavior change.

- Increase in students’ awareness (11)

- Positive behavior changes in students (6)

- Positive attitude changes in students (2)

- All students completed water usage log (1)

- More time is needed to see major behavior changes (1)

Question: Please suggest any improvements to the Storm Drain Rangers Program. N = 8

Results Summary: Although the feedback about this question was varied, three teachers did recommend shortening the length of the lessons while lengthening the duration of the program over the school year. Two teachers also commented that they were satisfied with the program and felt that it needs no improvement.

- Make the lessons shorter and spread them out over a longer period of time (3)

- Program is great how it is (2)

- Leave some materials in the classroom so teachers can review concepts (1)

- Get students to present more (1)

- Include more ways to engage English language learners (1)

- Visuals should be big and clear for whole class to see/reference (1)

- Add a field trip (1)

- Provide Spanish copies of anything sent home to students’ families (1)

- Number the pages in the curriculum binder (1)

Discussion and Conclusions

Overall, results indicate that the goals of the Storm Drain Rangers Program were achieved. The SDR Program provided meaningful hands-on watershed and storm drain education for elementary school students, and also provided professional development to classroom teachers in environmental education.

The goals of the Storm Drain Rangers Program included:

Students showed an overall increase in knowledge of the program content through results from the pre- and post-program surveys. The individual question results reveal that students did learn about their local watershed and its connection to the larger bay and ocean watersheds. Students also increased their awareness of the storm drain system and its connection to local waterways (i.e. creeks), the bay, and the ocean. An increase in knowledge about potential storm drain pollutants was also evident through the pre- and post-program surveys.

Both the student surveys and the teacher written evaluation forms show student comprehension of how their actions affect the local watershed environment and the animals and plants that share this environment. Students increased their awareness about the storm drain system and potential pollutants that can enter the system and affect the local creek, the bay, and the ocean. Many teachers commented that their students are more aware of how their actions can impact the environment. Teachers went on to report that their students have demonstrated a noticeable increase in environmentally-friendly behaviors, especially in the area of litter prevention (i.e. not littering anymore, telling others not to litter). A few teachers stated that their students’ attitudes about pollution and the environment have changed. These observations of positive changes in students’ awareness, attitudes, and behaviors are coming from teachers who are in close and constant contact with their students. This type of information about students is more valid coming from teachers than from students self-reporting about any changes they have made due to participation in the program. This is because the teachers serve as an ‘outside’ objective observer, whereas a variety of factors can come into play and lessen the validity of the observations if it comes from the students themselves.

Teacher survey and written evaluation form results indicate that the program provided professional development opportunities and the resources needed for teachers to feel comfortable teaching the SDR Program on their own the following school year. Our direct in-class training model, access to program equipment kit, and the comprehensive curriculum guide proved to be effective in preparing teachers to teach the program themselves. Every teacher responded that they plan to teach the SDR Program in their classrooms in the future.

The results to some survey questions also show how the program was a positive experience for teachers. Every teacher responded that they would recommend the program to other classroom teachers, which is an indication that they felt the program was valuable and worthwhile. All teachers also said that the program met their expectations. The program also inspired teacher participants to incorporate more hands-on science activities in their school-year curriculum.

We did receive a low number of completed evaluation forms and surveys from our teacher participants. Ideally, we would collect completed evaluation tools from all teacher participants so we could get a more fair and accurate sense of how the program has impacted teachers and students. Perhaps KIDS for the BAY staff should look into ways to ensure that more teachers complete and turn in the evaluation tools. For example, teachers could be given an incentive for turning in the evaluation tools in a timely manner.

Of the data we received from teachers who did complete a written evaluation form and surveys, the feedback and results were tremendously positive. This is a strong indication that the program is achieving its goals and providing vital professional development for teachers and hands-on environmental education to students that positively impacts both teachers’ and students’ awareness, attitudes, and behaviors towards the environment and storm drain pollution prevention.

Recommendations

Although overall we received encouraging results as to the impact of the Storm Drain Rangers Program on teachers and students, we do have a few recommendations to improve the program evaluation process itself in order to strengthen the process and better assess whether we are achieving our objectives.

And, finally

Appendices

Program Content and Activities Description
and
Evaluation Instruments

Appendix A. Storm Drain Rangers Content and Activities Description

Appendix B. Student Pre- and Post- Program Survey

Appendix C. Teacher Script for Administering Pre-Program Survey

Dear Teacher,

Thank you for helping us with our program evaluation by administering this survey to your class of students. Please read over the directions and carefully follow each direction when administering the survey to your class.

Directions

Before the Survey:

Please assign your students an identification number. Each student will need his/her own unique “Student ID #” and will need to use the same number for the pre- and post-program surveys. If students already have a number in the class (i.e. from the class roster or from the school), then have students use this number for their Student ID # on the surveys.

When Administering the Survey:

Say: “Our class will be doing a science program with KIDS for the BAY. We will be learning about our local environment and what we can do to make it a cleaner and healthier place for everyone.”

Say: “Before the program starts, each of you will fill out some information on a survey.” (Show them the survey.) “This survey is like a test, but you won’t be graded on your answers. KIDS for the BAY is asking us to fill this out because they want to find out what students learn through their programs.”

Say: “I will pass out the survey, and we will complete part of it together. Do not start on the questions yet.”

Pass out the survey and with your students complete the following sections on the top of each page: student ID #, date, teacher’s name, and school.

Say: “I will read each question out loud, and give you time to complete your answer. I will repeat the question if you need me to.”

Say: “You might not know how to answer some of these questions. It is okay if you don’t know the answer to a question. Just do your best. If you don’t know an answer, make your best guess.”

Read each question out loud, and then give students time to write their answer. Repeat the question if they need it read out loud again. Do not influence students’ answers at any point during the survey.

When students are finished, make sure students have their names and other information filled out on each page, collect all of the surveys, and put them in the KIDS for the BAY envelope. Give the envelope to your KftB Program Director during the next lesson.

Thank you again for helping us to improve our programs!

If you have any more questions, please do not hesitate to contact your KftB Program Director or Sheela Shankar, Education Director at KIDS for the BAY.

Appendix D. Teacher Script for Administering Post-Program Survey

Dear Teacher,

Thank you for helping us with our program evaluation by administering this survey to your class of students. Please read over the directions and carefully follow each direction when administering the survey to your class.

Directions

Before the Survey:

Your students will need to use the same unique identification numbers they used on their pre-program surveys. Please have these ID #’s ready so students can enter them onto their post-program surveys.

When Administering the Survey:

Say: “KIDS for the BAY wants to find out what you have learned through their program.” (Show them the survey.) “This survey is the same one you completed before the program started. The survey is like a test, but you won’t be graded on your answers. KIDS for the BAY is asking us to fill this out because they want to find out what you’ve learned.”

Say: “I will pass out the survey, and we will complete part of it together. Do not start on the questions yet.”

Pass out the survey and with your students complete the following sections on the top of each page: student ID #, date, teacher’s name, and school.

Say: “I will read each question out loud, and give you time to complete your answer. I will repeat the question if you need me to.”

Say: “You might not know how to answer some of these questions. It is okay if you don’t know the answer to a question. Just do your best. If you don’t know an answer, make your best guess.”

Read each question out loud, and then give students time to write their answer. Repeat the question if they need it read out loud again. Try not to influence students’ answers at any point during the survey.

When students are finished, make sure students have their names and other information filled out on each page, collect all of the surveys, and put them in the KIDS for the BAY envelope. Give the envelope to your KftB Program Director during the next lesson.

Thank you again for helping us to improve our programs!

If you have any more questions, please do not hesitate to contact your KftB Program Director or Sheela Shankar, Education Director at KIDS for the BAY.

Appendix E. Teacher Pre-Program Survey

KIDS for the BAY

Storm Drain Rangers Pre-Program Survey

Thank you for taking the time to complete this pre-program survey. The information you provide will be used to help us impr