HUXLEY COLLEGE OF THE ENVIRONMENT
Syllabus for Estu 575 (CRN 13843), 4 cr., Winter 2008
Assessment, Evaluation and Research in Environmental Education
Location: ES 345 Time: Tues & Thurs, 12:00-1:50pm
Instructor: Gene Myers, Ph.D.
Contact info: AH 224; x4775; ms 9085; mailbox in AH 217; gmyers@cc.wwu.edu
Office Hours: Tues 10-11; Wed 2:30-4:30. Sign up on office door preferred. Or arrange.
Web version: http://www.ac.wwu.edu/~gmyers/575W08.syl.htm
OVERVIEW AND AIMS:
This course will focus chiefly on program evaluation, with some attention to related topics in assessment and research. Evaluation is a critical component of the "R&D department" of educational programming. It has many faces and plays many roles, from needs assessment and "front-end" prototyping / piloting, to mission refinement, to determining the impacts of programs. To fulfill these roles, a diverse suite of skills is needed. This course will emphasize understanding the contexts of evaluation, as well as skills and approaches for a representative set of typical evaluation situations. Given the relative youth of the field of environmental education, and usual low funding for evaluation, it is perhaps not surprising that the evaluation of environmental education is still maturing. Nonetheless, it is rapidly growing, and the next generation of environmental education leaders needs to be well positioned to use evaluation to improve and promote the field. Therefore, this course plays a vital role in your graduate education in environmental education.
This course combines lecture, discussion, and an evaluation planning project to teach basic concepts and applied skills emphasizing program evaluation (and secondarily research and assessment), in the field of environmental education. You should come away from this course with a concrete appreciation of evaluation processes and products. After this course you should be able to bring a utilization-focused, “menu-driven” approach to evaluation wherever you find yourself, should be able to design program evaluations, particularly ones useful in program development, and know how to find and use additional resources when needed. This is a graduate level course, and as such expects a high level of effort and involvement by the student. The aims of this course are that the student will:
1. Understand how to shape evaluations to make them useful for intended uses, in a context- and user- sensitive manner.
2. Know basic approaches for the evaluation of typical EE situations and program types.
3. Become familiar with tools for generating and analyzing data used in evaluation.
4. Be able to evaluate evaluations according to standards of utility, feasibility, propriety, and accuracy.
5. Read and critique a variety of evaluation reports.
6. Prepare an evaluation plan for a specific environmental education program need.
REQUIREMENTS:
****In all assignments for this course, you should use the American Psychological Association’s citation format, as presented in its Publication Manual. A summary of its features is at: http://www.wisc.edu/writing/Handbook/DocAPA.html, but consult the actual handbook for precise details****
For some useful resources on writing and research, see: http://www.ac.wwu.edu/~gmyers/resstud.html; particularly see the links on academic honesty and plagiarism & related WWU policies.
EVALUATION:
15pts Class participation & discussion
25pts Written response to case materials or other readings
15pts Report on an EE evaluation report of your choice
45pts Evaluation plan
Grading breakdown (rounded up):
A = 100% - 93 %, A - = 92 - 90, B+ = 89 - 87, B = 86 - 83, B- = 82 - 80, C+ = 79 - 77, C = 76 - 73, C- = 72 - 70, D+ = 69 - 67, D = 66 - 63, D- = 62 - 60, F = 59 and below.
DISABILITIES ACCOMMODATION:
Any student with a disability that may affect their performance in this class is encouraged to speak to the instructor or the Office of Student Life (360-3844) to arrange for suitable accommodation.
TEXTS:
Required – as listed in schedule, below. They are found as follows:
1. Required readings, photocopied or provided by instructor on paper or electronically. Items listed in schedule are from these sources unless listed in web sources below.
2. AEA: American Evaluation Association materials, on line:
Principles, at: http://www.eval.org/Publications/aea06.GPBrochure.pdf;
Program Evaluation Standards, at: http://www.eval.org/EvaluationDocuments/progeval.html
What the Program Evaluation Standards Say about Designing Evaluations, at: http://www.wmich.edu/evalctr/jc/ (Click on the "Program Eval. Standards" book image, then on the line of text at top of page).
3. WWU’s Human Subjects Rights (IRB) pages http://www.wwu.edu/depts/rsp/human.html.
4. Wiltz, L. K. (2001). Proceedings of the Teton Summit for program evaluation in nonformal environmental education. Jackson Hole, WY: Teton Science School & Ohio State University. Available at saldata or: http://www.ag.ohio-state.edu/~eetap/publications.htm (scroll down to link near bottom of page)
5. Thompson, G. & Hoffman, J. (n.d.) Measuring the success of environmental education programs. Calgary, Alberta: Canadian Parks and Wilderness Society. Available at saldata or: http://www.green-street.ca/providers/documents/Measuring_the_Success.pdf
6. W. K. Kellogg Foundation Logic Model Development Guide. Available at saldata or: http://www.wkkf.org/pubs/tools/evaluation/pub3669.pdf
7. Trochim, William M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW page, at URL: <http://www.socialresearchmethods.net/kb/> (version current as of 20 Oct. 2006). (See the “Navigating” section to familiarize yourself.)
Supplemental:
- UFH: Stevens, F., Lawrenz, F. & Sharp, L. (2002). User-friendly handbook for project evaluation: Science, Mathematics, Engineering and Technology Education. Washington DC: National Science Foundation. Available free at: http://www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf
- Frechtling, J. & Sharp, L. (1997). User-Friendly Handbook for Mixed Method Evaluations. NSF. http://www.nsf.gov/pubsys/ods/getpub.cfm?nsf97153
- Several excellent supplemental readings are free on the web through AEA: http://www.eval.org/ (see Links; Publications; Discussion lists, etc.)
- http://www.be-webbed.com/evalResources.html
- Other excellent sources are noted in bibliographies of assigned readings.
SCHEDULE (subject to change):
|
Wk/Day/Date |
Topics |
Readings & Assignments |
|
1T Jan. 8 |
Intro to course, overview of evaluation, research and assessment. Major types & uses of evaluation Professional practice standards |
-Rsch v Eval v Assmt; Types, purposes & methods of evaluation; More on goals; Evaluation process (photocopy) -AEA Principles& Standards (electr) -WWU Human Subjects Rights (electr) -Trochim: foundations/ethics (electr) |
|
1R Jan. 10 |
Bridging evaluation and non-formal EE. |
-Wiltz (Teton Summit) (electr)-PARKS evaluation (from instructor) -NAAEE Nonformal guidelines (saldata) -Due: Response to question given in class. |
|
2T Jan. 15 |
Outcomes monitoring & evaluation |
-Thompson & Hoffman, esp. 19-28; & Appendix One; skim rest (electr) -Kellogg Fndn Logic Model Guide (electr) |
|
2R Jan. 17 |
ZATPAC Needs Assessment / logic models / Teen program / Empowerment eval case |
-ZATPAC case materials, starting with ZATPAC case2.doc (saldata) Due: Response to question on ZATPAC case -Due: draft Logic Model for program of your choice (draft Indicators=extra credit) |
|
3T Jan. 22 |
No class - day off to make up for Sun. Jan 27 field trip |
|
|
3R Jan. 24 |
Observational methods |
-Evaluation of exhibits (photocopy) -Observational methods & Diamond, Observational tools (photocopy) -Birjulin & Saunders, Summative evaluation of Living Coast exhibit (photocopy) -recommended: WPZ Spiderworld (saldata) |
|
Sunday, Jan 27 All day |
Fieldtrip to Woodland Park Zoo. Host: Kathryn Owen. Others guests: Susan Clayton; Kris Morrisey & her UW grad students |
|
|
4T Jan. 29 |
Observational methods Debrief WPZ visit |
-Evaluation of exhibits (photocopy) -Observational methods & Diamond, Observational tools (photocopy) -Birjulin & Saunders, Summative evaluation of Living Coast exhibit (photocopy) -recommended: WPZ Spiderworld (saldata) |
|
4R Jan. 31 |
Recycling Ed evaluation Case Part A |
-Part A & docs (photocopy) -3rd.xls spreadsheet (saldata) -Due: Response to question at end of Part A |
|
5T Feb. 5 |
Recycling Ed evaluation Case Part B
|
-Part B & docs (photocopy) -Diamond Ch 7 presenting data (photocopy) -Patton Ch. 3 (photocopy) -Due: Response to question at end of Part B |
|
5R Feb. 7 |
Recycling Ed evaluation Case Part C |
-Part C & docs (photocopy) -Patton ch. 4 & 5 (photocopy) -Due: Response to questions for Part C |
|
6T Feb. 12 |
Evaluation research questions and methods |
-Types of research; Qualities of good evaluation questions; Research designs for impact; Sampling (photocopies) -Trochim: Foundations/Language of Research (all); Sampling (all subsections); Design (all subsections of Internal Validity, then main sections through Quasi-experimental designs. Read first 2 subjections under Experimental, plus covariance; under Quasi-exper., read non-equivalent design. More if you want.) |
|
6R Feb. 14 |
Data & measurement |
-Validity and reliability of measurement; Writing test items; Quantitative instruments in EE citation list; -Brody- concept map ex (2 pgs) (photocopy) -Diamond, Ch 6 Measuring Learning (photocopy) -Trochim: Measurement (read all up to (not including) Survey Research, plus Scaling) |
|
7T Feb. 19 |
Analysis |
-Trochim: Analysis (all subsections up to Inferential Statistics—there, read sections on T-test, Gen Linear Model, Posttest only, and Covariance) |
|
7R Feb. 21 |
Evaluation of evaluation report presentations |
Due: Write-up of your evaluation. |
|
8T Feb. 26 |
Qualitative methods - 1 |
-Qualitative methods (photocopy) -Trochim: Measurement/ Qualitative measures -Monroe (photocopy) -Notes on interviewing children (photocopy) -Myers, Saunders & Garrett (saldata – large!) |
|
8R Feb. 28 |
Qualitative methods - 2 |
-Fabiano: Top Ten misconceptions of focus group research, and Moderating focus groups (photocopy) -Hillcoat (saldata) |
|
9T Mar. 4 |
Social Marketing Eval Case Part A |
Background on watershed protection |
|
9R Mar. 6 |
Social Marketing Eval Case Part B |
-Social marketing /survey methods case study (photocopy) -Jull & Simpson & example questionnaires and phone protocol (photocopy) -Jull, Colkitt & Cantrell
(saldata) |
|
10T Mar. 11 |
Social Marketing Eval, cont |
-Trochim: Measurement / Survey research (all) -Survey and Question design; guidelines for creating better questionnaires (photocopy) -optional excellent resource: |
|
10R Mar. 13 |
Project presentations |
|
|
Mon Mar 17, 1-3p |
Official final time (or TBA) Project presentations |
|
Estu 575 Win 2008 - Evaluation Plan Assignment
Work individually or with a small group to develop an evaluation plan for some environmental education program, or aspect thereof, as appropriate to program life-cycle and other needs.
Elements of an Evaluation Plan in the abstract:
The elements of an evaluation plan in general include some variation on the following, depending on your guiding evaluation question(s):
1) A careful, balanced, succinct description of the program to be evaluated, including its goals and context. The developmental stage of the program (how many years it has been running, what kinds of evaluation have been used already, etc.) should be discussed so that you can explain the appropriateness of the type of evaluation you propose.
2) A characterization of the type of evaluation planned, and a statement of the evaluation question(s) to be answered. The question should exhibit the qualities of a good evaluation question. It should address intended users and uses, and how you determined these (i.e., document stakeholder interviews, negotiations with primary stakeholders, and how these eventuated in the question(s) to be answered); it might well make reference to AEA Utility standards.
3) Evaluation design. This part of the plan lays out the methods by which data will be obtained that will answer the evaluation question(s). In general the elements, for each evaluation question, include the following:
a) Unit(s) of analysis (agencies, students, classes, teachers, seminars, conflicts, etc.).
b) Focal concerns (in the case of qualitative evaluation questions) and/or dependent and independent variables (what you will gather data on). Please note that front end, needs assessment, evaluability and other types of non-summative evaluations are fully legitimate, depending on the context… and the evaluation should be tailored accordingly.
c) Study design. If relevant, the critical comparisons (i.e., pre/ post; 'control' groups), and how you propose to control for possible confounds or threats to validity.
d) Sampling plan (who, where, how many, how accessed, permissions, HSR procedures). Include plans for dealing with possible response biases.
e) Data gathering tools (interviews, surveys, tests, observations, other instruments), including description of how data will be physically recorded. A draft of each instrument, ideally refined by some piloting, should be included.
f) Times of measurement or data gathering for each technique.
g) Anticipated analyses, including scoring or other reduction procedures and specific comparisons or other patterns of findings.
The above list will require modification depending on your questions. An evaluation question that is best answered by qualitative methods may require a different set of considerations than those above. The bottom line is: The description of the design elements should make it clear how the data you propose to gather will answer the evaluation questions. In other words, if you can trace from the anticipated form of the presentation of results back to the guiding question and uses and defend each step based on accuracy standards, you have a good plan.
4) A brief description of the proposed write-up including the anticipated scope of conclusions and recommendations (what kinds of statements you will and will not be able to make based on your data), and how they will be presented so as to facilitate the intended uses.
5) A proposed time line that fits the client’s actual timeline. Completing this step serves as an important feasibility check on the plan.
6) An estimated budget broken-down by person-hour time (not money) estimates for each work step in the plan. Hours should be broken down by staff, contracted-out, or in-kind, depending on skills required and their availability.
7) Citations of relevant literature (and a References list at end). Such literature may include theory, previous similar studies, and methodological authorities upon which your design draws. Use APA style.
Grading
Your work will be graded based on the merits of your overall plan and design, with emphasis on how adequately you have foreseen and addressed possible pitfalls, within the scope of material covered in the class. The final product should reflect professional level of writing competence.