KIDS for the BAY - Environmental Action Through Education

Program Evaluation Reports - Four Rs Action Program

Four Rs Action Program 2006-07
Evaluation Report

Executive Summary

Introduction
This evaluation of the Four Rs Action Program was conducted to assist the KIDS for the BAY (KftB) staff and program partners with determining the impact and effectiveness of their in-depth waste reduction program on teacher and student participants, and to a lesser degree students’ families. The evaluation also serves as an invaluable tool in informing future programming and future program evaluation. Thirty elementary school teachers and over 950 students participated in the Four Rs Action Program during the 2006-07 school year. Fourteen teachers also participated in the second year follow-up program during the 2006-07 school year. These teachers received support, resources, and guidance as they implemented the program themselves with their students. This executive summary focuses on highlights and conclusions of the results from the evaluation process. A full report follows the summary.

The goals of the Four Rs Action Program are:

1. Thirty teachers learn to use the Four Rs message as an educational resource to stimulate students’ learning.

2. Nine hundred students increase their awareness of the issues of resource conservation and waste reduction and change their behaviors to help conserve resources and reduce waste.

3. The Four Rs message is integrated into the curriculum and culture of our target schools. Principals, teachers, students and parents learn the value of reducing, reusing, recycling and composting and practice these behaviors in their schools and at home.

To gather the data needed to assess whether these goals and objectives were met, KftB administered a pre- and post-program survey and a written evaluation form to all teacher participants (first year teachers and second year follow-up teachers). Eighteen teachers completed pre- and post- program surveys and nineteen teachers completed written evaluation forms. Eight second year follow-up teachers completed a pre- and post- program survey as well.

One-hundred-and-three students completed pre- and post-program surveys that checked to see if students’ increased their knowledge around solid waste management, resource conservation, the storm drain system, and the Four Rs. These students were in classes that were at the fourth grade level and had relatively low numbers of English Language Learners. Results from all of these evaluation tools are included in this report.

Results: Highlights and Conclusions
Overall, results indicate that the goals of the Four Rs Action Program were achieved. The Four Rs Action Program provided professional development for teachers and meaningful, exciting, hands-on education for students in solid waste management and the Four Rs message.

Students showed an overall increase in knowledge of the program content through results from the pre- and post- program surveys. The individual question results reveal that students learned key vocabulary terms and learned about solid waste management, resource conservation, the storm drain system, and the Four Rs.

Both first and second year teachers shared that their students’ awareness of and concerns about solid waste management increased as a result of the program. A few teachers reported a positive change in attitude towards solid waste management, not littering, and practicing the Four Rs.

Even more encouraging was feedback from teachers about an increase in Four Rs behaviors in their students. Most follow-up teachers reported that their students increased waste reduction behaviors as a result of participating in the program. First year teacher participants wrote many examples of how their students engaged in Four Rs behaviors during and after the program.

Teacher participants increased their comfort level in teaching about solid waste issues, teaching about the Four Rs, and facilitating a waste reduction action project with their students due to participation in the program. This increase in comfort, and therefore perceived ability, in these areas will contribute to the teachers continuing to teach the Four Rs Action Program in future years.

Many teachers reported that they gained professional development and increased their teaching skills through the in-class modeling of the program. A few teachers appreciated learning how to connect children to science in a way that is relevant to their lives. Teachers also reported an increase in their own knowledge about solid waste management and the Four Rs as a result of the program.

All teachers felt prepared to teach the Four Rs Action Program to their students the following school year. Teachers felt that the resources we provided to them (curriculum guide, in-class modeling, and equipment kit) contributed to their ability to teach the program lessons and activities. Follow-up teachers remained confident teaching about the Four Rs, facilitating a waste reduction action project, and teaching the Four Rs Action Program in their second year.

Teachers shared the ways that their school communities had been influenced by the program. Many teachers reported that the Four Rs Action Program helped to reinforce, and in some cases increase, the school’s efforts to recycle and reduce waste. A few teachers also said that their students completed projects through the program that educated the students’ schoolmates and the school community about the Four Rs message.

Teachers shared examples of how students’ families have been influenced by the program. Many teachers reported that the Four Rs Action Program has contributed to waste reduction behaviors and an increase in awareness about the Four Rs in students’ homes. A few teachers felt that, because they do not regularly interact with their students’ families, they could not properly assess the impact the program has had on students’ home environments. KIDS for the BAY could work to improve the evaluation process to better assess the impact the program has on students’ families.

Given the results from the gathered evaluation data, KftB concludes that program goals and objectives were met and that we should continue to provide the program to elementary school teachers and students. The evaluation process also enlightened us on ways to improve the evaluation tools themselves in order to acquire relevant and useful evaluation data.

Four Rs Action Program 2006-07
Evaluation Report

Project Overview

Project Summary
The Four Rs Action Program (Four Rs Program) provides in-depth, hands-on learning experiences focused on reducing waste and conserving resources for third, fourth, and fifth grade students. Classroom teachers also learn alongside their students and are provided resources and support to continue teaching the program themselves.

The goals of the Four Rs Action Program are

1.Thirty teachers learn to use the Four Rs message as an educational resource to stimulate students’ learning.

2.Nine hundred students increase their awareness of the issues of resource conservation and waste reduction and change their behaviors to help conserve resources and reduce waste.

3.The Four Rs message is integrated into the curriculum and culture of our target schools. Principals, teachers, students and parents learn the value of reducing, reusing, recycling and composting and practice these behaviors in their schools and at home.

Program Description
The Four Rs Action Program includes five two-hour workshops at the school site and hands-on mini-action projects within the classroom lessons that effectively reduce waste and conserve resources. In addition, each class selects a waste-reduction action project, which the students plan, develop and implement with guidance from their teacher and support and resources from KIDS for the BAY (KftB).

Teachers learn the programs alongside their students, attend planning and evaluation meetings with KftB Instructors, and teach preparation and follow up activities from the Four Rs Action Program curriculum guide. Each team of teachers receives an equipment kit to continue teaching the program. KftB follows up with teachers in the year after their training to provide additional support and to evaluate success as the teacher teaches the program to her students.

A detailed description of the Four Rs Action Program content and activities is located in Appendix A.

Evaluation Goals

KIDS for the BAY’s program evaluation process is both summative and formative. The evaluation data collected from both teacher and student participants during the 2006-07 school year enables us to assess 1) the impact of the program on students and teachers, and 2) the effectiveness of the program content and delivery and how we can improve our teaching methods and curricula for future programming. An added benefit to the evaluation process is being able to assess the effectiveness of the evaluation tools themselves in gathering the desired data from program participants. Information on the types of responses we received, the quality of the responses, and the amount of data we collected from each evaluation tool we administered in 2006-07 will allow us to improve the tools themselves as well as the overall evaluation plan.

Specifically, the Four Rs Action Program evaluation process seeks to inform us of the following:

Program Effectiveness and Improvement

1. Are we reaching our stated program goals and objectives for teacher and student participants?

2. How can we improve the Four Rs Action Program based on the feedback collected from teachers and the results of the evaluation process?

Student Participants

1. What was the students’ overall experience of the program?

2. Was there any increase in student knowledge due to participation in the program?

3. Were there any changes in students’ attitudes, abilities, or behaviors due to participation in the program?

4. What was the impact of the program on students’ families? In particular, have students’ families become more aware of local environmental issues and/or engaged in environmentally responsible behaviors as a result of the program?

Teacher Participants

1. What was the teachers’ overall experience of the program?

2. What suggestions do they have to improve the program content and delivery?

3. Were there any increases in teachers’ comfort level and perceived ability in:

teaching environmental science concepts

teaching about solid waste issues

teaching about the Four Rs- reduce, reuse, recycle, and rot

facilitating a waste-reduction action project with their students?

4. Do teachers feel prepared to teach the program next year?

5. How useful were the various program components (in-class modeling, curriculum guide, equipment kit) in providing teachers with what they need to teach the program?

Methods

Quantitative and qualitative evaluation tools were administered to teacher and student participants between September 2006 and June 2007. The Four Rs Action Program (intervention) consisted of five in-class workshops and a waste-reduction action project. Thirty 3rd- 5th grade classroom teachers and over 900 students participated in the Four Rs Program during the 06-07 school year. Fourteen teachers also participated in the second year follow-up program during the 2006-07 school year. These teachers received support, resources, and guidance as they implemented the program themselves with their students.

The evaluation data from teachers was collected from the entire participant group. The evaluation data from students was collected from a sample of the entire participant group. The methodology behind each evaluation tool varies and is described separately for each tool below.

Student Pre- and Post- Program Surveys:
A sample of students completed a pre-survey (Appendix B) before the first classroom workshop, and completed an identical survey within one month of the completion of the classroom workshops. Out of the thirty Four Rs Action Programs, we selected classes that were at the fourth grade level and that had the lowest number of English Language Learners in order to create a more uniform pool of students. Then we randomly selected classes of students from this pool to complete the surveys.

The educational objectives for each classroom workshop were used as the basis for developing each question on the survey. The surveys were designed to show whether the lesson objectives were met and whether there were any changes in students’ knowledge as a result of participating in the Four Rs Program. The surveys contained a variety of question-and-answer styles and techniques (i.e. multiple choice, fill-in-the-blank, pictograms) that are appropriate and suitable for the age of the student participants (8-11 year olds). A variety of questioning strategies were used. Some questions simply checked for knowledge while others required critical thinking skills and/or more depth of knowledge. The survey contained a total of thirteen questions.

One-hundred-and-three students completed the pre- and post-program surveys. Each classroom teacher administered the surveys using a script (Appendix C and D) to introduce and facilitate the survey process. Students used identification numbers instead of their names to identify themselves on their surveys. The anonymity of the surveys helped students to understand that this was not a “test” that would be included in their school grades.

Teacher Pre- and Post-Program Surveys:
Each of the thirty teacher participants completed a short pre-program survey (Appendix E) in September-October 2006, before the program started. They then completed a longer post-program survey (Appendix F) that contained the same questions as the pre-program survey, plus additional questions. This survey was administered at the completion of all program activities, including the action project. The pre-post survey questions contained standard likert-scale response options and asked teachers about their comfort level in various areas such as facilitating a waste-reduction action project with their students and teaching environmental science concepts. The additional questions included in the post-survey asked how the resources we provided and the program structure will enable them to teach the program themselves. Results for the pre- and post-program surveys reflect the data collected from any of the thirty teacher participants who completed and turned in both surveys. Eighteen teachers completed both surveys, and three additional teachers completed just the post-survey with additional questions.

Teacher Written Evaluation Forms:
Each teacher participant completed written evaluation forms (Appendix G) that asked open-ended questions about their overall impressions of the program, program highlights, suggestions for program improvement, and how the program has impacted them, their students, and the school community. Teachers completed one written evaluation form at the completion of the classroom workshops, and completed another at the close of all program activities. The open-ended questions allowed teachers to be insightful in their responses and also allowed them to be detailed in their answers. Nineteen teachers completed the written evaluation form; the results from these nineteen teachers are included in this report.

Follow-Up Teacher Post-Program Surveys:
Each second-year follow-up teacher participant was asked to complete a post-program survey (Appendix H) in June 2007, after program activities had been completed. The post survey questions contained standard likert-scale response options and asked teachers about their level of confidence in various areas, such as teaching about the Four Rs. The survey also asked teachers to rate how the program impacted their students. Some questions also asked how the resources we provided (i.e. curriculum guide, support from KftB) helped teachers to teach the program themselves. Results for the post-program surveys reflect data collected from eight follow-up teachers who completed and turned in the survey.

Data Analysis

Student Pre- and Post Program Surveys:
Each question on the survey was given a specific point score (weight) and the total point score for the survey (13 questions) equaled 100. Some questions were given a higher weight than others based on what we felt was important for the students to know and understand. In other words, we prioritized the concepts and information we wanted students to know and assigned weight to each survey question accordingly. If we felt if was important for the students to know a certain concept, the correlating question was given a higher weight than other questions on the survey.

Each class who completed the surveys was given a class code to identify them. We then compiled the pre-program and post-program surveys for each student using their identification numbers and discarded any surveys that did not have both a pre- and a post program survey. Each pair of surveys was then given a new identification code. This code was recorded on both the surveys and the excel spreadsheet. The pre-surveys were graded and the results for each question were recorded in an excel spreadsheet. We also calculated the total point score for each survey and put this in a separate column. The post-surveys were graded and the results were entered into a separate excel spreadsheet.

Using the software program XLSTAT, we compared the pre-survey results to the post-survey results using a paired t-test. The survey results were compared to see if there was a significant increase in students’ knowledge due to participation in the Four Rs Action Program.

Teacher Pre- and Post-Program Surveys:
The pre-survey and post-survey responses from each teacher were put into a table and compared to see what, if any, changes occurred as a result of the intervention. The changes in responses were put into a separate table and then compiled into three percentage statements (increase, no change, or decrease) for each survey question. These statements are shown in the Results section. Results are shown for all Four Rs Action Program teacher participants that completed and turned in both pre- and post-program surveys. Eighteen teachers completed both surveys, and three additional teachers completed just the post-survey with additional questions. The responses to the questions that were asked only in the post-program survey were compiled and displayed in a separate table.

Teacher Written Evaluation Forms:
We reviewed the nine questions posed on the written evaluation form and selected a few key questions to formally analyze. The qualitative data we received from teachers’ written evaluation forms was then categorized and quantified. We read and re-read all of the responses to each of the selected questions, eventually drawing out common themes and categories that emerged from the responses. As the responses were repeatedly reviewed, we merged some categories and created sub-categories where needed. As responses were placed into the final categories, we took note of how often each type of response was made. The results for the written evaluation forms are displayed using this information. The response category is listed first, and then any specific statements that were made that related to the category are listed below. The number in parentheses next to the category/statement indicates the number of responses made for that category/statement. Nineteen teachers completed the written evaluation form; the results for each selected question from these nineteen teachers are included in this report.

Follow-Up Teacher Post-Program Surveys:
The post-program survey responses from each follow-up teacher were put into a table. Data was collected from the eight follow-up teachers that completed the post-program survey. Results for each question were compiled and displayed by the frequency and percentage of each type of response. The response with the highest frequency for each question was highlighted.

Results

Student Pre- and Post-Program Survey Results

Whole Test Results

Results Summary: Paired t-test results from 103 fourth grade students determined that there was a statistically significant increase in knowledge after experiencing the Four Rs Action Program intervention (t102 =9.97, p<0.0001). The total possible score for the entire test (13 questions) was 100 and the mean score increase between pre- and post-tests was 16.49 points.

Pre-Test Mean Score	Post-Test Mean Score	Mean Score Increase
63.08	79.57	16.49

Individual Question Results

Question 1 asked students to write each of the Four Rs in order of the hierarchy of effectiveness. Partial points were given for recording all of the Four Rs; complete points were given to students who wrote the “R’s” in the correct order.

Results Summary: Results show a statistically significant increase in knowledge about the names of the Four Rs and their relative hierarchies.

Question	Total Possible Score	Paired t-test Results	Mean Score Increase
1	10	t(102) = 8.55,p<0.0001	1.78

Questions 5 and 9 checked students’ knowledge about landfills. Question 5 checked to see if students knew that the solid waste stream ends at the landfill. Question 9 determined whether students knew that solid waste remains the same once it is deposited into a landfill environment.

Results Summary: Results show a statistically significant increase in knowledge about the solid waste stream and landfills.

Question	Total Possible Score	Paired t-test Results	Mean Score Increase
5	10	t(102) = 9.84,p<0.0001	3.22
9	8	t(102) = 3.16,p<0.001	1.20

Questions 2 and 13 checked for students’ knowledge about recycling. Question 2 asked students to identify which of the listed items can be placed in a recycling bin. Question 13 checked to see if students knew the three steps necessary to ‘close the loop’ in recycling.

Results Summary: Results show a statistically significant increase in knowledge about steps to “close the loop” in recycling and the types of materials that can be put into a recycling bin.

Question	Total Possible Score	Paired t-test Results	Mean Score Increase
2	9	t(102) = 0.57,p<0.284	0.13
13	6	t(102) = 3.01,p<0.002	0.80

Questions 3 and 4 focused on the storm drain system. Question 3 checked to see if students knew that storm drains connect to the local creek (for Alameda schools, the storm drains connect directly to the San Francisco Bay). Question 4 asked students to identify from a list the types of actions that can cause storm drain pollution.

Results Summary: Results show a statistically significant increase in knowledge about the storm drain system and potential storm drain pollutants.

Question	Total Possible Score	Paired t-test Results	Mean Score Increase
3	4	t(102) = 2.24,p<0.014	0.37
4	9	t(102) = 4.91, p<0.0001	1.31

Question 6 checked to see if students knew the definition of the first of the Four Rs, “reduce”.

Results Summary: Results show a statistically significant increase in knowledge about the definition of the term “reduce” as it relates to the Four Rs.

Question	Total Possible Score	Paired t-test Results	Mean Score Increase
6	8	t(102) = 4.82,p<0.0001	1.94

Question 7 listed seven personal actions and asked students to identify which “R” the action could be categorized as. The question checked to see if students understood ways to practice each of the Four Rs.

Results Summary: Results show a statistically significant increase in knowledge about how to classify waste reduction actions into each of the Four Rs.

Question	Total Possible Score	Paired t-test Results	Mean Score Increase
7	7	t(102) = 6.71,p<0.0001	1.06

Question 8 checked to see if students knew the definitions of the major vocabulary words presented in the program lessons.

Results Summary: Results show a statistically significant increase in knowledge about the definitions of common waste reduction/solid waste management terminology.

Question	Total Possible Score	Paired t-test Results	Mean Score Increase
8	7	t(102) = 7.09,p<0.0001	1.39

Question 10 asked students to respond to a true/false question about the connection between natural resources and the Four Rs. The statement was: “If we practice the Four Rs, then we can conserve, or save, natural resources.”

Results Summary: Results show a statistically significant increase in knowledge about the connection between engaging in waste reduction behaviors and conserving natural resources.

Question	Total Possible Score	Paired t-test Results	Mean Score Increase
10	10	t(102) = 1.09,p<0.139	.49

Question 11 checked for students’ knowledge of the external anatomy of a red wiggler worm.

Results Summary: Results show a statistically significant increase in students’ knowledge of the external anatomy of a red wiggler worm.

Question	Total Possible Score	Paired t-test Results	Mean Score Increase
11	4	t(102) = 5.34,p<0.0001	.87

Question 12 asked students to identify items that can be put into a compost bin. Students were given eight choices of common lunch leftovers which included organic and inorganic items.

Results Summary: Results show a statistically significant increase in knowledge of the types of waste that can and cannot be put into a compost bin.

Question	Total Possible Score	Paired t-test Results	Mean Score Increase
11	8	t(102) = 2.86,p<0.003	1.94

Teacher Pre- and Post-Program Survey Results

Question 1: To what extent do you feel comfortable teaching about local solid waste issues? N = 18
Results Summary: The majority of teachers felt more comfortable teaching about local solid waste issues after participating in the Four Rs Action Program.

16.5% of teachers felt less comfortable after participating in the Four Rs Action Program
22% of teachers felt the same, no change, after participating in the Four Rs Action Program
61.5% of teachers felt more comfortable after participating in the Four Rs Action Program

Question 2: To what extent do you feel comfortable teaching about the Four Rs- reduce, reuse, recycle, and rot? N = 18
Results Summary: The majority of teachers felt more comfortable teaching about the Four Rs after participating in the Four Rs Action Program. A considerable amount of teachers did not change their level of comfort in teaching about the Four Rs.

5.5% of teachers felt less comfortable after participating in the Four Rs Action Program
39% of teachers felt the same, no change, after participating in the Four Rs Action Program
55.5% of teachers felt more comfortable after participating in the Four Rs Action Program

Question 3: To what extent do you feel comfortable facilitating a waste reduction action project with your class? N = 18
Results Summary: The majority of teachers felt more comfortable facilitating a waste reduction action project with their class after participating in the Four Rs Action Program.

17% of teachers felt less comfortable after participating in the Four Rs Action Program
22% of teachers felt the same, no change, after participating in the Four Rs Action Program
61% of teachers felt more comfortable after participating in the Four Rs Action Program

Question 4: To what extent do you feel comfortable teaching environmental science concepts? N = 18
Results Summary: The majority of teachers felt more comfortable teaching environmental science concepts after participating in the Four Rs Action Program. A considerable amount of teachers felt the same or less comfortable teaching environmental science concepts.

22.5% of teachers felt less comfortable after participating in the Four Rs Action Program
33% of teachers felt the same, no change, after participating in the Four Rs Action Program
44.5% of teachers felt more comfortable after participating in the Four Rs Action Program

Questions 5 – 8: Post-Program Questions
Results Summary: All classroom teachers feel prepared to teach the Four Rs Action Program themselves after participating in the program, with the majority of teachers feeling prepared ‘to a considerable extent’. All teachers felt that the resources we provided them enable them to teach the program themselves, with the ‘in-class modeling’ having the highest ratings. The ‘curriculum guide’ and ‘equipment kit’ had a strong high percentage rating of enabling teachers to teach the program themselves ‘to a considerable extent’. A few teachers did not respond to question 8 because they had not received the equipment kit at that time and therefore did not know how to respond.

N=21	To No Extent	To a Slight Extent	To a Moderate Extent	To a Considerable Extent	To a Great Extent	No Answer
5) To what extent do you feel prepared to teach the Four Rs Action Program to your class next year?			5 24%	9 43%	7 33%
6) To what extent do you feel the curriculum guide we provide enables you to teach the program?		3 14%		11 52%	6 29%	1 5%
7) To what extent do you feel the in-class modeling we provide enables you to teach the program?				8 38%	12 57%	1 5%
8) To what extent do you feel the equipment kit enables you to teach the program?				9 43%	7 33%	5 24%

Teacher Written Evaluation Form Results

Question: How has the Four Rs Action Program helped you as a classroom teacher?
N = 19

Results Summary: Six teachers reported that the resources we provide them help them to teach the program themselves. Six teachers also commented that their teaching skills/abilities increased due to participating in the program. Four teachers reported an increase in personal knowledge about waste issues and waste reduction strategies. The remaining comments were varied. There were no negative comments in this section.

Results:

1. KftB provided resources to teach the program themselves (6)

Provided lessons to use year after year (2)

Provided ability to teach the program herself due to resources and modeling (1)

Provided science curriculum to incorporate Four Rs into the school curriculum (1)

2. Increased teaching skills, professional development for teachers (6)

Provided ways to connect children to science relevant to their lives (3)

Covered science standards (1)

KftB Instructor modeled how to teach program content in a practical, meaningful, engaging way (2)

KftB Instructor modeled how to motivate the class and engage students in learning (1)

Well-constructed lessons were presented with skill and enthusiasm; instructor was a role model and an inspiration (1)

Provided teaching strategies to teach the content (1)

Program demonstrated the effectiveness of hands-on learning (1)

Increase in teachers’ knowledge of solid waste, the Four Rs (4)

Validated environmental discussions held in classroom (2)

Increase in teachers’ Four Rs behaviors (2)

Increased awareness about personal actions to reduce waste (1)

Increased interest in and behavior changes towards the Four Rs (1)

Program raised teachers’ consciousness about practicing the Four Rs (2)

Progam increased teacher’s ability to integrate environmental issues into all curriculum areas (2)

Program enabled teacher to see how students respond to/interact with guest teacher (2)

Program provided meaningful instruction within NCLB/scripted learning climate (1)

Program validated importance of individual behaviors regarding the environment (1)

Guest instructor gives teacher a break (1)

Question: Please share the impact that the Four Rs Action Program has had on your students. Have you noticed a change in attitude or behavior in your students as a result of the program? N = 19

Results Summary: The majority of teachers reported an increase in their students’ awareness about solid waste issues and the Four Rs as a result of the program. Eight teachers also wrote that their students’ have shown an increase in Four Rs behaviors due to program participation. The remaining results were varied, but reflect positive outcomes of the program on the students.

Results:

11. Increase in students’ awareness about solid waste issues and Four Rs (15)

Students are thinking more about waste (2)

Students are thinking about the positive impact they can make by practicing Four Rs (2)

Students having more discussions about waste issues and the Four Rs (2)

Program provided long-term, sustained learning for students (1)

Program concepts and the Four Rs were explained to students, students were not just told what to do (1)

Students interested in taking care of environment (1)

Students can apply knowledge learned through program in every day life (1)

Students realize their actions affect environment (1)

12.Increase in Four Rs behaviors as a result of program (8)

Students are recycling more, better (5)
- Increase in paper recycling (2)

Students are reusing more (4)

Students are throwing less trash on playground (2)

Students have expressed behavior changes at home (i.e. now recycling) (2)

Students not wasting as much paper (1)

Students bringing lunches from home (1)

Students using worm bin, composting food waste (1)

13. Increase in positive attitudes towards the Four Rs in students (2)

14. Program had bigger impact on students than any other waste reduction program (2)

Length and depth of program beneficial to students (1)

15. Students teaching others to practice Four Rs (2)

Students teaching schoolmates (1)

Students teaching their families (1)

16. Students can better discuss environmental issues (i.e. pollution, trash disposal) (2)

17. Students loved the program, KftB Instructor (1)

18. Instruction was fun and interesting for students (1)

19. Action project solidified impact they can have (1)

20. Students empowered to improve environment (1)

21. Program contributes to long-term behavior change and attitude change in students (1)

Question: How has the program impacted your students’ families and/or the school community? N = 19
Results Summary: Six teachers reported that the program educated their students’ families about reducing waste through practicing the Four Rs. Five teachers wrote that their students’ families increased their waste reduction behaviors as a result of the program. Four teachers felt unsure about the impact the program had on students’ families. Five teachers felt the program reinforced the school’s efforts to incorporate Four Rs practices. Three teachers reported that their students spread the Four Rs message to the school community through informational posters, reuse art displays, and presentations.

Results:
Program Impact on Students’ Families

22. Program educated families about the Four Rs (6)

Program raised the awareness about what can and cannot be put in the trash (2)

Family pledges informed and involved families in program (1)

Informational Fair about the Four Rs educated families (1)

Program influenced the way families shop at the grocery store (1)

Reuse art projects taught families about reusing (1)

23. Families engaged in Four Rs behaviors as a result of the program (5)

Increase in recycling efforts(4)

Increase in practicing the Four Rs (1)

24. Unsure about the impact on families at this time (4)

25. Students shared fun and excitement of program with families (1)

26. Families mentioned appreciating the opportunity to participate in program (1)

27. Students are now more involved in family activities, i.e. gardening (1)

Program Impact on the School Community

28. Program reinforced/increased the school’s efforts to recycle and reduce waste (5)

The school started a recycling program (1)

29. Students spread the Four Rs message to schoolmates/the school community (3)

Through Four Rs posters (2)

Through Four Rs art projects (1)

Through teaching younger students (1)

Follow-Up Teacher Post-Program Survey Results

Results Summary: Results show that in their second follow-up year, all teachers felt confident teaching about the Four Rs, with the majority of teachers ‘strongly agreeing’. All teachers also felt comfortable teaching the program themselves. Seven out of eight teachers felt that their students’ waste reduction behaviors increased due to participation in the program. The majority of teachers (seven out of eight) also felt that the program has led to their students’ increased concern about solid waste issues.

N = 8	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree	Not Applicable
I feel confident teaching about the Four Rs—reduce, reuse, recycle, and rot.				2 25%	6 75%
I feel confident facilitating a waste reduction action project with my class.				4 50%	4 50%
Participation in the Four Rs Action Program has increased my students’ concern about solid waste issues.			1 12.5%	3 37.5%	4 50%
Participation in the Four Rs Action Program has increased waste reduction behaviors in my students.			1 12.5%	4 50%	3 37.5%
The curriculum guide provided to me was helpful in teaching the program myself this year.			1 12.5%	3 37.5%	4 50%
The support from my KftB Program Instructor increased my confidence in teaching the program myself this year.				4 50%	4 50%
The program equipment kit provided to me was helpful in teaching the program myself this year.				7 87.5%	1 12.5%

N = 8	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree	Not Applicable
I feel confident teaching the Four Rs Action Program myself.				5 62.5%	3 37.5%
I plan to continue teaching the program in my classroom.			1 12.5%	3 37.5%	3 37.5%	1 12.5%

Discussion and Conclusions

Overall, results strongly indicate that the goals of the Four Rs Action Program were achieved.

The goals of the Four Rs Action Program included:

1. Thirty teachers learn to use the Four Rs message as an educational resource to stimulate students’ learning.

2. Nine hundred students increase their awareness of the issues of resource conservation and waste reduction and change their behaviors to help conserve resources and reduce waste.

3. The Four Rs message is integrated into the curriculum and culture of our target schools. Principals, teachers, students and parents learn the value of reducing, reusing, recycling and composting and practice these behaviors in their schools and at home.

Program Impact on Students
Students showed an overall increase in knowledge of the program content through results from the pre- and post-program surveys. The individual question results reveal that students learned about the issues surrounding solid waste management and actions they can take- the Four Rs- that reduce waste and conserve resources. An increase in knowledge about key vocabulary terms and the hierarchy within the Four Rs was also evident. Students showed that they learned how to recycle and compost more efficiently. Students also learned about the storm drain system and how pollutants such as garbage can travel through this system to a local creek or bay habitat.

Students also increased their awareness of and concern for the issues of resource conservation and waste reduction. Many teachers reported that they noticed a marked increase in awareness about solid waste and the Four Rs in their students. Second year follow-up teachers also shared that their students’ concerns about solid waste increased as a result of the program. A few teachers reported a positive change in attitude towards solid waste management, not littering, and practicing the Four Rs.

Both first year and second-year follow up teachers went on to report that their students have demonstrated a noticeable increase in Four Rs behaviors. These observations of positive changes in students’ awareness, attitudes, and behaviors are coming from teachers who are in close and constant contact with their students. This type of information about students is more valid coming from teachers than from students self-reporting about any behavior changes they have made due to participation in the program. Classroom teachers can make objective observations of their students, instead of the students themselves making claims about their own behaviors.

Each class of students also completed a waste-reduction action project which the students planned and implemented with guidance from their classroom teacher and a KftB Instructor. The action project provided an opportunity for students to engage in Four Rs behaviors and make a positive difference in their communities.

Program Impact on Classroom Teachers
Teacher participants increased their comfort level in teaching about solid waste issues, teaching about the Four Rs, and facilitating a waste reduction action project with their students due to participation in the program. This increase in comfort, and therefore perceived ability, in these areas will contribute to the teachers continuing to teach the Four Rs Action Program in future years.

Beyond increasing their comfort level in these areas, teacher participants learned how to teach the Four Rs to their students in a fun and engaging way. Many teachers reported that they gained professional development and increased their teaching skills through the in-class modeling of the program. While teaching the program to students, KftB Instructors showed classroom teachers how to lead the lessons, discussions, and hands-on activities. A few teachers appreciated learning how to connect children to science in a way that is relevant to their lives. Teachers also reported an increase in their own knowledge about solid waste management and the Four Rs as a result of the program.

All teachers felt prepared to teach the Four Rs Action Program to their students the following school year. Teachers felt that the resources we provided to them (curriculum guide, in-class modeling, and equipment kit) contributed to their ability to teach the program lessons and activities. Feedback about the usefulness of the provided resources was given in both the post-program survey and in teachers’ written evaluation forms.

Results from surveys administered to second-year follow-up teachers are extremely positive. Follow-up teachers remain confident teaching about the Four Rs, facilitating a waste reduction action project, and teaching the Four Rs Action Program in their second year. The majority of follow-up teachers also felt that the resources provided to them in their second year, including support from KftB, helped them to implement the program with their class. Most teachers plan on continuing the program in their classrooms.

KftB does note that we received a lower amount of completed written evaluation forms and pre- and post-program surveys from teachers than expected. We could improve the way we administer the evaluation forms and surveys, and improve the way we communicate with our classroom teachers, to ensure that we receive a higher amount of completed evaluation tools.

Program Impact on Students’ Families and the School Community
Teachers shared the ways that their school communities had been influenced by the program. Many teachers reported that the Four Rs Action Program helped to reinforce, and in some cases increase, the school’s efforts to recycle and reduce waste. A few teachers also said that their students completed projects through the program that educated the students’ schoolmates and the school community about the Four Rs message.

School principals also signed contracts in the beginning of the school year which showed the principal’s commitment to and support of the Four Rs Action Program. KftB Instructors held meetings with many principals at our partner schools. At these meetings, KftB Instructors collected verbal feedback from principals and included this feedback in their program reports. We received very positive feedback from school principals. Many expressed their enthusiasm for the program, and a few shared ways that the program has supported and/or influenced their schools’ efforts to reduce waste.

Recommendations

We have received very encouraging and positive results from the evaluation process. The data collected from teachers and students shows how we have met the goals of the Four Rs Action Program. We do however have a few recommendations to improve the program evaluation process itself in order to strengthen the process and better assess how we are achieving program goals and objectives.

1. Continue to provide hands-on, engaging environmental science lessons to elementary school students. Continue to provide meaningful and relevant learning experiences and opportunities for students to be engaged in learning about and caring for their environment.

2. Continue to provide classroom teachers with in-class training, resources such as a comprehensive curriculum guide and program equipment, and support so teachers feel comfortable and confident to teach about the Four Rs and teach the program activities/lessons.

3. Evaluate and continue to improve the program evaluation process, which will in turn strengthen the program goals, objectives, content, and delivery.

And, finally
4. Evaluate and continue to improve the evaluation tools used to assess the impact of the program on teacher and student participants. The evaluation process is cyclical in nature, in that we are constantly moving along a cycle of assessment, program improvement, and evaluation improvement. Current recommendations to improve the evaluation tools include:

Teacher Written Evaluation Forms and Pre- and Post-Program Surveys; Follow-Up Teacher Written Evaluation Forms and Pre- and Post-Program Surveys: Improve the way KftB Instructors communicate to teachers about the importance of completing and turning in the evaluation tools. Encourage KftB Instructors to be persistent with following up with teachers who have not completed the evaluation tools, to ensure that we receive the completed tools in a timely manner. Entertain the possibility of providing an incentive to teachers for completing the tools and turning them in to us on time.

Family Evaluation Tool: Include an evaluation tool and/or additional questions to existing tools to assess the impact of the program on students’ families, and in particular any behavior changes due to their children’s participation in the program.

Appendices

Program Content and Activities Description
and
Evaluation Instruments

Appendix A. Overview of Four Rs Action Program Lessons and Action Projects

Lesson One: Urban Run-Off Pollution and Neighborhood Clean Up
Students learn about the storm drain system and the connection between the school neighborhood and the local creek, bay, and ocean. Students then complete a neighborhood clean up and categorize the garbage collected into items that can be recycled, reused and composted. They begin to analyze where the garbage in their neighborhood is coming from and how it can be reduced at its source. Students also take home their surveys to share the information with their families. Families then make a pledge together to reduce urban run-off pollution.
Service-learning project: Neighborhood survey and clean up

Lesson Two: Landfills and Our Waste Issues
Students learn the different methods of waste disposal and the problems associated with disposing of waste. They work in groups to create mini-landfills in the classroom, bury garbage, and predict what will happen to the garbage when they excavate it. Students learn about the hazards of waste incineration and about environmental justice issues both locally and internationally.

Teacher’s Pre-Lesson: Natural Resources and the Four Rs
Classroom teachers lead this lesson as a precursor to the lessons that focus on the Four Rs. The class defines what natural resources are. Students choose an object commonly found in the classroom, identify the materials used to make the product, and trace those materials back to the natural resources they were made from. Students then make a hanging mobile that illustrates what they have learned. Students understand the connections between our use of natural resources, the Four Rs, and the amount of waste we create. The remaining lessons on reduce, reuse, recycle, and rot build on the concepts learned in this lesson.

Lesson Three: Reduce and Reuse
Students study packaging and discuss the benefits and drawbacks to packaging. They share a picnic in the classroom and analyze the packaging used for common lunch items. Students learn about the power of consumer and personal choices, and how to make smarter choices that reduce excessive waste. Students also compare single-use items to items that can be used again, and discuss ways they can incorporate reusable items into their daily routine. Students apply what they’ve learned about resource conservation to what they are learning about reducing and reusing. Students complete an art project out of garbage and recycle materials. This is a homework assignment that students complete with their families to involve their families in what they are learning.
Service-learning project: Reuse art project from garbage and recycling materials

Lesson Four: Recycle
Students learn about “closing the loop”, and the resources and energy saved by not only recycling paper, plastic, aluminum, and glass, but also by buying items made from recycled materials. Students analyze a variety of plastic materials to discover if they can or cannot be recycled in their city. Students make recycled paper using scrap paper from
Appendix A.

the classroom, thereby reducing classroom waste and conserving resources. Students also make colorful informational posters that illustrate factual information about garbage and recycling and post them around the school or in the windows of local businesses to encourage others to practice the Four Rs.
Service-learning project: Recycled paper from classroom paper scraps; informational posters that teach others waste and recycling facts.

Lesson Five: Rot (Compost)
Students learn about the natural process of decomposition. Students then learn about the F.B.I. (fungus, bacteria, and invertebrates) and their role in breaking down organic waste. Students investigate a worm compost bin and learn how composting reduces waste. They learn that organic waste can be turned into a valuable resource by composting. Interested classes have the option of setting up a worm compost bin in their classroom to compost organic waste from their snacks and lunches. Students also take home an interview sheet and interview a family member about solid waste management and the Four Rs. The student and the family member record pledges to practice the Four Rs at home.
Service-learning project: Option of setting up a worm composting system in the classroom or school garden

Student-Designed Action Projects
Students have the opportunity to apply what they have learned about the Four Rs and resource conservation by choosing, planning, and implementing an action project. The classroom teacher provides guidance to the students and a KftB Instructor provides guidance, support, and materials if necessary. Examples of action projects that classes have designed and implemented include: a Four Rs Faire to teach families and school staff about the Four Rs message; a class reuse art contest and display; informational brochures to dispense information to the school community about the Four Rs; a letter-writing campaign to a company or school administrator; and letters to a family member or friend to persuade them to incorporate Four Rs practices into their daily routine.

Appendix B. Student Pre- and Post-Program Survey

KIDS for the BAY

Four Rs Action Program
Student Survey

Student ID#___________________ Date_______________________________
Teacher’s Name____________________ School _____________________________

1) What are the Four Rs? Write each of the Four Rs in order below.

R_____________ R_____________ R_____________ R______________

2) What can you put in a recycling bin at home?

Put a check-mark next to each thing that can be put in a recycling bin.
_ Sheets of newspaper
_ A plastic toy
_ An aluminum soda can
_ A battery

Appendix B.

Student ID#___________________ Date________________________________
Teacher’s Name______________________ School______________________________

3) The storm drains around your school connect directly to:
(check off only one answer)
_ the Pacific Ocean
_ the sewer
_ the local creek
_ the Atlantic Ocean
_ the San Francisco Bay

4) Put a check mark next to each thing that can cause storm drain pollution.
_ washing a car in the street with soap
_ pouring motor oil in the street
_ a car with smoke coming out of the tail pipe
_ pouring paint in the kitchen sink
_ throwing garbage on the street

5) Where does our garbage end up after it is thrown in a garbage can?

Put a check-mark next to your answer below.
_ In the sewer.
_ In the local creek.
_ In the landfill.
_ At the dump.

Appendix B.

Student ID#___________________ Date________________________________
Teacher’s Name______________________ School______________________________

6) What does the Four Rs term “reduce” mean?

Put a check-mark next to your answer below.
_ To put something away.
_ To make less garbage.
_ To make something new.
_ To use something again.

7) Francis practices the Four Rs all the time. Below are some of the things he did last
week to make less waste. Put the correct “R” next to each thing that Francis did to
recycle, reuse, rot (compost), and reduce.

Recycle Reduce Rot Reuse

__Recycle_ Recycled a plastic water bottle.
_________ Saved something to use it again instead of throwing it in the garbage.
_________ Used both sides of a piece of paper.
_________ Put grass clippings in the green waste bin at home.
_________ Bought something that was made from recycled materials.
_________ Brought his own bag to the grocery store.
_________ Took only 1 paper napkin at a restaurant.
_________ Put an apple core in his classroom worm compost bin.

Appendix B.

Student ID#___________________ Date________________________________
Teacher’s Name______________________ School______________________________

8) Match each word with its definition. Draw a line from each definition to the correct
word.

Recycle

Compost

Reuse

Reduce

Landfill

Natural Resources

Waste

Another word for garbage, what we throw away.

A place where our garbage ends up after its thrown in a garbage can.

Things that are found in nature such as water, trees, and minerals that we use to make things.

To turn something old into something new.

To use something again.

To buy less and make less trash.

What worms and other organisms turn old food into.

9) What do you think happens to garbage once it is in a landfill?

Put a check-mark next to your answer below.
________ The garbage disappears.
________ The garbage stays the same.
________ The garbage gets eaten by animals and turns into compost.
________ The garbage turns into liquid.

Appendix B.

Student ID#___________________ Date________________________________
Teacher’s Name______________________ School______________________________

10) Is the following statement true or false?

If we practice the Four Rs, then we can conserve, or save, natural resources.
_ True
_ False

11) Use the vocabulary words to label the body parts of a Red Wiggler Worm.

setae prostomium anus clitellum

12) Kristina just finished her lunch at school. She has some things left over.

Put a check mark next to each thing from Kristina’s lunch that can be composted.
________ Bread
________ Plastic spoon
________ Banana peel
________ Aluminum foil

Appendix B.

Student ID#___________________ Date________________________________
Teacher’s Name______________________ School______________________________

13) What are the three actions needed to “Close the Loop” in recycling?

Put a check mark next to each of the three actions.
________ Tell a friend to recycle.
________ Old materials get made into new products.
________ Put recyclable materials into a recycle bin.
________ Refill your plastic water bottle.
________ Buy products made from recycled materials.

Appendix C. Teacher Script to Administer Student Pre-Program Survey

Dear Teacher,

Thank you for helping us with our program evaluation by administering this survey to your class of students. Please read over the directions and carefully follow each direction when administering the survey to your class.

Directions

Before the Survey:
Please assign your students an identification number. Each student will need his/her own unique “Student ID #” and will need to use the same number for the pre- and post-program surveys. If students already have a number in the class (i.e. from the class roster or from the school), then have students use this number for their Student ID # on the surveys.

When Administering the Survey:
Say: “Our class will be doing a science program with KIDS for the BAY. We will be learning about our local environment and what we can do to make it a cleaner and healthier place for everyone.”
Say: “Before the program starts, each of you will fill out some information on a survey.” (Show them the survey.) “This survey is like a test, but you won’t be graded on your answers. KIDS for the BAY is asking us to fill this out because they want to find out what students learn through their programs.”
Say: “I will pass out the survey, and we will complete part of it together. Do not start on the questions yet.”
Pass out the survey and with your students complete the following sections on the top of each page: student ID #, date, teacher’s name, and school.
Say: “I will read each question out loud, and give you time to complete your answer. I will repeat the question if you need me to.”
Say: “You might not know how to answer some of these questions. It is okay if you don’t know the answer to a question. Just do your best. If you don’t know an answer, make your best guess.”
Read each question out loud, and then give students time to write their answer. Repeat the question if they need it read out loud again. Do not influence students’ answers at any point during the survey.
When students are finished, make sure students have their names and other information filled out on each page, collect all of the surveys, and put them in the KIDS for the BAY envelope. Give the envelope to your KftB Program Director during the next lesson.

Thank you again for helping us to improve our programs!

If you have any more questions, please do not hesitate to contact your KftB Program Director or Sheela Shankar, Education Director at KIDS for the BAY.

Appendix D. Teacher Script to Administer Student Post-Program Survey

Dear Teacher,

Directions

Before the Survey:
Your students will need to use the same unique identification numbers they used on their pre-program surveys. Please have these ID #’s ready so students can enter them onto their post-program surveys.

When Administering the Survey:

Say: “KIDS for the BAY wants to find out what you have learned through their program.” (Show them the survey.) “This survey is the same one you completed before the program started. The survey is like a test, but you won’t be graded on your answers. KIDS for the BAY is asking us to fill this out because they want to find out what you’ve learned.”

Say: “I will pass out the survey, and we will complete part of it together. Do not start on the questions yet.”

Pass out the survey and with your students complete the following sections on the top of each page: student ID #, date, teacher’s name, and school.

Say: “I will read each question out loud, and give you time to complete your answer. I will repeat the question if you need me to.”

Say: “You might not know how to answer some of these questions. It is okay if you don’t know the
answer to a question. Just do your best. If you don’t know an answer, make your best guess.”

Read each question out loud, and then give students time to write their answer. Repeat the question if they need it read out loud again. Try not to influence students’ answers at any point during the survey.

When students are finished, make sure students have their names and other information filled out on each page, collect all of the surveys, and put them in the KIDS for the BAY envelope. Give the envelope to your KftB Program Director during the next lesson.

Thank you again for helping us to improve our programs!

If you have any more questions, please do not hesitate to contact your KftB Program Director or Sheela Shankar, Education Director at KIDS for the BAY.

Appendix E. Teacher Pre-Program Survey

Please answer the following questions:

To what extend do you feel comfortable teaching about local solid waste issues?