What is EVALUATION?
You do not need special training in research or evaluation methods
in order to conduct successful and useful evaluations. Evaluation planning
often helps to support program planning and development. Evaluation results
can also help to inform and improve grant programs as they develop. Evaluations
do not focus only on whether the programs "worked" or "didn't
work”. Evaluations are ongoing assessments of the progress toward meeting
grant missions, objectives and goals.
There are many different ways to conduct evaluations, and professional evaluators
tend to agree there is no "one best way" to do any evaluation. Good
evaluations should provide useful information about program functioning that
contributes to program improvement. Good evaluation requires carefully thinking
through:
- the questions that need to be answered
- the type of program being evaluated
- the ways in which the information generated will be used
This chain of events links inputs to outputs and outputs to outcomes in response
to a situation which articulates what your grant hopes to achieve and how.
In order to accomplish this broader view of evaluation, you must think about
such questions such as:
- What are the risk and protective factors to be addressed? (the goals)
- What services and activities will be provided? (the strategies)
- Who will participate in or be influenced by the grant program? (the target group)
- How will these activities lead to expected outcomes?
- What immediate changes are expected for individuals, organizations, or communities?
- What changes would the grant program ultimately like to create?
Evaluative measures should come from the mission, goals, objectives of the
grant with an emphasis on achieving measurable impacts. Impacts evaluated are
usually clustered into Outputs and Outcomes.
OUTPUT measures are quantifiable indicators of the amount of services delivered,
or clients served. These measures can be obtained through counts such as the
number of clients served, the number of service units (e.g., hours, courses
or sessions) delivered.
OUTCOME measures are quantifiable indicators of the benefit resulting from
the delivery of client services, or delivered services. These evaluative measures
show the impact the grant program has had on participants. The best OUTCOME
measures show that people completing the program have improved the behavior,
knowledge, skills, or attitudes targeted by the grant program.
How do I PLAN for an evaluation?
The first step should be when to plan for your evaluation. Begin planning
your program's evaluation at the same time you start the grant process. It
is crucial to plan for evaluation from the very beginning of each grant year
in the specific areas the program was designed to target. Use continuous measures
that can be repeated over the scope of the project. Provide balanced evaluations
utilizing both outputs and outcome measurements to report progress.
The next step is to decide: what questions do you want to answer? The cornerstones
of any good evaluation are the evaluation questions. What are you trying to
find out by doing the evaluation? In order to formulate your evaluation questions,
first consider the following issues:
- What is the purpose of the evaluation?
- When do you need the information?
- What resources do you have available for evaluation?
In order to plan your evaluation, you will need to address the following items:
-
What is going to be evaluated?
The first question that you need to answer is what you want to evaluate.
This may not be easy depending upon how clearly defined your program is.
For example, you might want to evaluate a comprehensive prevention program
that involves student education, community awareness, and information dissemination.
Think about the program's purpose and content. Do you want to examine the
whole program or just a particular component of it? Do you want to examine
all activities or just selected ones? Often, it isn't necessary to evaluate
all components of a comprehensive program.
-
Which program elements are most likely to have the strongest effects on participants?
For example, a one-shot motivational speaker is not likely to have significant
long-lasting influence on youth attitudes. However, a year-long sequence of
classes is likely to have a strong influence.
-
Which program elements require the greatest proportion of your resources?
Aspects of grant programs that consume high proportions of resources might
be more important to evaluate.
-
Feasibility of evaluation.
Some prevention activities are likely to be extremely difficult to evaluate,
such as media campaigns that reach thousands of homes, or awareness marches.
Your evaluation resources might be better used to evaluate other aspects of
programming for which evaluation will not be as difficult or costly.
How do I IDENTIFY MEASURABLE INDICATORS?
A challenge in any evaluation is identifying the information that best answers
the question(s). Indicators express that which you wish to know. They are the
measures or observable evidence that answer your questions.
Sometimes the needed information is obvious and straightforward (e.g., number
and demographic profile of clients served, or description/number of services
delivered). At other times the desired information is not specific enough to
collect directly. For example, a short-term outcome question of interest might
be:
- Did the program improve the quality of the relationship between parents
and their children? "Quality of relationship" is a vague concept.
You will need to define these terms before you can evaluate them:
- What does a positive parent-child relationship look like?
- How would you see it?
- How can you measure it?
It may also be important to understand how participants themselves define
and understand key program outcomes.
Consider this example: Program staff defined
and evaluated the outcome of a bilingual parent education program for Hispanic
parents as knowledge gained (child development knowledge gained, proficiency
in language). An evaluation showed little, if any, gains in knowledge. Upon
further probing, it was found that the participants were very satisfied with
the program. For them, it had been very successful because at its conclusion
they were able to share child-rearing tips with other parents and discuss
the influence of American culture on their children. Parents felt that they had
developed a support network of other parents that they could go to for advice
and assistance in child-rearing. Staff-defined definitions of outcomes missed
some important benefits as perceived by the participants.
It is usually important to engage different people with various backgrounds and cultural
understandings in the identification of indicators. "Quality Relationship" for
example, has different meanings for different people in different places. Listening
to multiple perspectives will produce indicators that are more complete and
meaningful. Likewise, key stakeholders may have particular ideas about indicators
that measure certain phenomenon. Your program will want to pay attention
to these ideas if the results are to be used by these stakeholders. You may
even wish to ask a few key individuals what evidence they think would answer the
question.
What evaluation METHODS will I use?
The decision concerning which methods to use for your grant evaluation involves
consideration of a number of factors. The W. K. Kellogg Foundation makes the
following recommendations:
- Stay focused on the questions that you want to answer and select methods
that are most appropriate to answering that question. Don't select your method
before selecting your question!
- Consider available resources. Usually, there is more than one way to collect
information about any particular question. Some methods involve more time,
money, and effort than others. Make realistic decisions about methods depending
on your resources.
- Be sensitive to the types of participants in your project and the kinds
of information you are collecting. Some information may place participants
at risk if it becomes public (e.g., any reporting of illegal activity). Other
information may be sensitive to some people or groups; be sure and consider
how respondents will feel if they are asked to provide particular information.
You may also need to consider the particular skills of your respondents. If
most don't read English, for instance, you will need to provide alternative
ways of obtaining the information.
- Credibility: How credible will your evaluations be as a result of the methods
that you have chosen? Consider the following:
- Is the instrument valid? That is, does it measure what it claims to measure?
- Do the questions make sense for your program?
- Is the instrument reliable? That is, will it provide the same answers if administered at different times or places?
- Are the methods suitable for the target group being studied?
- Are the methods biased toward finding only positive results?
What types or SOURCES OF INFORMATION are available?
There are many different types of information that can be collected about
the program. The following are some of the most common.
-
Written surveys or questionnaires
A written document containing questions answered by an individual. Survey
questions can be open-ended (e.g., "What did you like about this program")
or close-ended (e.g., "How much did you like this program? (check one):
____ not at all ____ a little ____ somewhat _____ a lot”).
-
Interviews
Interviews involve one person asking questions of another person. Interviews
can be done face-to-face or by telephone. Interviews, like surveys, can involve
open-ended questions and close-ended questions. Interviews can be used in many
different ways in an evaluation.
-
Tests and assessments
Tests and assessments are usually tools that have already been developed for
particular purposes. They can provide information about different aspects of
the evaluation but are most frequently used for assessing short- and long-term
outcomes. Tests can include physical assessments, knowledge or achievement
tests, psychological tests, and attitudinal assessments. Tests and assessments
might be included as part of written surveys, questionnaires, or interviews.
-
Observations
Observations of program activities are often used to understand the program
context. Observational techniques can also be used to collect some forms of
outcome data.
-
Focus groups
Focus groups involve asking a series of predetermined questions to a group
of people. People are encouraged to provide their opinions, but not necessarily
to reach consensus. Focus groups can be used in a variety of ways in evaluation.
-
Document and program record review
Existing program records and documents (such as meeting minutes, reports,
brochures, participation, etc.) can provide excellent information, particularly
about program context and history, and about program implementation. Case files
often contain a wealth of information about program participants that can be
useful in the evaluation. Attendance records and staff logs often contain information
about how many people have received services and how many services have been
delivered.
-
Confidentiality
On of the most challenging effort is obtaining securing data sources for evaluative
measures. These sources can range from complex to very simple depending on
the grant parameters.
- Individual data does not have to be collected in order to provide
information. Data might be requested by group (e.g., for these x names, please
return an overall absentee rate) without requesting name by name information.
- Information can be reported for selected groups provided that within
the measure, precise information is provided concerning who is
included and who is not included in the subgroup population
(e.g., of 100 participants only 20 are included in this measure).
-
Existing databases and archives
Many kinds of data are collected by public agencies, such as the Census Bureau,
school districts, police departments, and other county, state and federal agencies.
These data can be an important source of information for evaluation (see confidentiality
section above).
How do I evaluate my OUTCOMES?
In assessing your grant program, you want to know whether the grant program
had any effect.
- Did the program do what it was intended to do?
- Did the program achieve its stated goals and bring about the desired changes?
Program evaluators must use whatever data they have, but also plan what
data will and can be available for the evaluation. When establishing a plan
for evaluation at the beginning of each grant year, a balance of
both output and outcome measures should describe the activities of your grant
objectives. There are many kinds of measures that can be utilized to describe
unique programs.
You can utilize measures already available or kept on file to respond to evaluation
requirements or you may develop new measures if appropriate to your program
grant. Several common frameworks include looking at the number of types of
services provided to clients or measuring knowledge, skills, attitudes and
behaviors through pre-post tests or post-test only tests. We can examine how
to evaluate by looking at:
Data Collection Measures
There are many different methods for collecting outcome evaluations. Below are
some of the most common evaluation methods, including:
-
Post-test only data collection
Often, program outcomes are measured only after the program is completed.
This is understandable since programs must first be developed and operated
as planned before they can be assessed. Although collecting data only after
the program is implemented can't tell you how much participants have changed
(because you don't know what their status was before the program), this information
can contribute additional information to the description of your local program
and to the overall picture.
Outcomes measured only after a program is completed provide you with information
about where your participants stand at one point in time. You may learn, for
instance, that students in your drug information program have mastered 85 percent
of the knowledge about the effects of alcohol and other drugs as measured by
a drug information test.
What do you do with outcomes measured only after the program when there are
no appropriate published standard rates of knowledge, attitudes, use, or program
outcomes? Often, we need to have some basis of comparison before concluding
that the program brought about a change compared to how things were before
the program.
-
Pre-Post data collection
The most direct way to know if the prevention program changed program participants'
knowledge, attitudes, behavior, or some other outcome is to test program participants
before the program and again after the program. Comparing the difference between
before-program scores with after-program scores (e.g., after-program scores
minus before-program scores) will clearly indicate if a change on the outcome
scores has occurred. Students will, for example, have increased in drug knowledge
or decreased in accepting attitudes toward drug and alcohol use.
-
Measuring client satisfaction
Although not technically a program outcome or output, it is often very important
to learn about whether or not persons participating in programs are satisfied
with the services they have received. This can be very useful information for
making improvements to the program in the future.
-
Changes in annual/periodic rates
Many projects are required to collect data on an ongoing basis in relation
to number of participants/clients served, as well as various project specific
information data (such as length of time participants spent in a grant project
or program, number of referrals given, crime rates, recidivism, etc.). These
data are available to analyze the change in annual or periodic rates and do
not need to be recollected again.
Conclusion
The underlying purpose of any evaluation is to promote understanding and learn
about programs, and the ultimate result is to articulate what we learned about
the program, our professional competencies, and the evaluation process.
If you have any additional questions or comments regarding the determination
of appropriate outputs or outcomes, please contact the PPRI office.
|
