Week 11 Assignment

Review of Chapters 1-10

Chapter 1:  What Is Evaluation?

There are two definitions that define evaluation.  The first, evaluation is the systematic process of collecting and analyzing data in order to determine whether and to what degree objectives have been or are being achieved.  The second is evaluation is the systematic process of collecting and analyzing data in order to make a decision.  The evaluation process includes examining the efficiency, effectiveness, and impact of the program.  Using alternative options such as comparing one activity to another or one program to another to make a decision.  An evaluator needs to identify areas to improve and other levels of evaluation.  This includes Kirkpatrick’s four levels of evaluation (Level 1:  Reaction, Level 2:  Learning, Level 3:  Behavior, Level 4:  Results).  The essential parts of the design format may include evaluation questions, activities, data sources, population sample, data collection design, responsibility, and data analysis.  

Chapter 2:  Why Evaluate?

There are long and short term benefits of an evaluation.  One of the biggest benefits includes being able to answer the “why” to program sponsors and staff.  Evaluations can provide opportunities to identify new audiences or to identify audiences that should no longer use the program.  The benefits of evaluation provide an increased knowledge of outcomes.  With the benefits there are also limitations.  Evaluations do not guarantee change.  Evaluations may affect behavior, but not necessarily performance.  Before deciding to evaluate, social and political situations need to be considered.  Decision-making needs and research for new solutions may provide further clarification of the “why”.

Chapter 3:  Decision Making:  Whom to Involve, How, and Why?

The traditional program cycle starts with an organization’s mission and goals.  Then the cycle moves to needs analysis, then to program planning, program implementation where formative and summative evaluation takes place.  In the needs analysis phase, the evaluator’s job is to help clarify the goals, assist in identifying the need, and help make a link between the program’s mission and the identified needs.  The program planning phase, the method for achieving the objectives are selected.  The next phase is the program implementation phase.  This is where formative evaluation takes place.  At the end of the evaluation, the final stage is considered the summative stage.  

Chapter 4:  Starting Point:  The Evaluator’s Program Description

This chapter focuses on the Evaluator’s Program Description or the EPD.  According to Boulmetis & Dutwin, an EPD is a statement prepared by the evaluator after working with stakeholders to amplify and clarify all aspects of the program, including goals, objectives, activities, and anticipated outcomes.  An EPD reveals a program’s goals and objectives, which activities are planned to accomplish the goals and objectives, and what measurement tools are in place.  The information the EPD provides will assist you later in developing an evaluation design format.  

Chapter 5:  Choosing an Evaluation Model

There are four popular evaluation models.  One is the discrepancy model which aims to understand the evidence to prove a cause-and-effect relationship.  The goal-free model is another.  In this model, the program goals are not part of the criteria.  Instead, the evaluation looks at how and what the program is doing to address the needs of the client population.  The transaction model is goal-based and combines monitoring with process evaluation through a continuous back-and-forth between the evaluator and staff.  The Decision-Making Model use used to make a decision about the future use of the program.  Other models include the systems analysis model, the art criticism model, the adversary model, and the goal-based method.  After a method is chosen, then the evaluator can begin to the overall evaluation design format.  The format introduces the components that may occur, but do not always all occur, in any evaluation.  Those components include evaluation questions, program objectives, activities observed, data sources, population samples, data collection design, responsibility, data analysis, and audience.    

Chapter 6:  Data Sources

There are two techniques used to collect data.  They are quantitative and qualitative.  Quantitative refers to a numerical data and qualitative refers to finding meaning through verbal narrative and observations.  There are four levels of data which include nominal data, ordinal data, interval data, and ratio data.  First, the evaluator should check for any existing data.  Then collecting new data should begin.  The tools used to collect data are interviews, scales, sentence completion, tests, and observational analysis.

Chapter 7:  Data Analysis

The evaluator will determine the most appropriate method of data analysis for each objective.  Data usually refers to the numerical figures that are presented in a table or graph.  However, data can also be take other forms such as narratives, scales, and/or counts.  Key terms from the chapter include statistics, measures of central tendency (mode, mean, and median), measures of variability (ranges, quartiles, percentiles, and standard deviation and variance.  There are different levels of data which include nominal, ordinal, interval, and ratio.  Ratio data is seldom dealt with.  

Chapter 8:  Is It Evaluation or Is It Research

Research and evaluation are different.  If the main intention of what you are doing is to examine and better understand the processes that occurred during the program cycle, then you are performing an evaluation.  If the main intention of what you are doing is to better understand the outcomes of the processes, then you might be performing evaluation, research, or evaluation-research.  Finally, if the main intention of what you are doing is to better understand the long-term effects and what is known in the professional field, then you are performing research.  In research, there is an assumption that all variables can be controlled, variables can be manipulated, and establishing the cause-and-effect relationship which is unlike evaluation.  The chapter also discusses different sampling methods-probability sampling and nonprobability sampling.  

Chapter 9:  Writing the Evaluation Report

After an evaluation is complete, most evaluations require an evaluation report that summarizes the goals, the methodology of the evaluation, findings, interpretations, conclusions, and recommendations.  A summative evaluation report requires supporting tables, graphs, charts, or case studies that address targeted results.  There are seven sections that should be included in the evaluation report.  Section 1:  Summary, Section 2:  Statement of the Evaluation’s Purpose, Section 3:  Background Information, Section 4:  Description of the Evaluation Study, Section 5:  Results, Section 6:  Discussion of the Program and Its Results, and Section 7:  Conclusions and Recommendations.

Chapter 10:  Evaluation as a Business

There is a market for Program Evaluation.  There are opportunities in the public sector and in the private sector.  In order to be able to make evaluation a full-time business you will need to have a string of several evaluation contracts.  If you are new to the evaluation field, taking a proactive track is the best option.  Establishing an area of expertise is suggested.  Once you decide if you want to work within the public or private sector, you can narrow your search for proactive resources available.  There are also reactive and emergency resources available.  Before you can negotiate your contract, you need to know what your time, expertise, and work are worth.  Once you have done that, you can negotiate your responsibilities and prepare a contract.  In order to keep your business alive as an evaluator, you must deliver on what you promised.   

Extra Credit:

I had Week 11 Assignments pulled up on one tab and the pdf file that was sent out earlier in the semester.  I was mistakenly looking at the pdf file for this week’s extra credit opportunities and not the current Week 11 assignments.  The extra credit options are different and I didn’t realize that until just now.  I spent quite a bit of time taking the quiz and am still going to include it here even though I now realize that you did not offer it as an option.  I got an 87 even though I got more answers correct.  You must have to answer with the exact words because my answers were correct, but if I added anything other than the way the correct answer was originally entered in, it was marked wrong.  I understand you more than likely won’t count it, but I am putting it here anyway since I spent time on it.

Optional Bonus #1

To be submitted for the weekly assignment: Go back and review chapters 1-10 in the Boulmetis & Dutwin text.  Be sure each chapter writing is specifically attuned to the particular chapter and not something generic. Maintain focus on the chapter you are writing about. Remember to designate your responses by chapter (e.g., 1, 2, 3 …) so it is obvious which writing goes with which chapter.  Instead of writing a narrative response, please use one of the following sites (no substitutions without prior approval) and create a presentation of your summaries.

Optional Bonus #2

For this week’s discussion, choose one chapter from the book that has been the most helpful.  Please give a thorough reason as to why it was helpful (ex. clarified a specific part of the Evaluation Report Project and/or RFP).  In addition to a chapter that was helpful, post any questions that you may have about a specific chapter or any specific content from the book that you need clarified.