Evaluation Methodology 101

Types of Evaluation

Formative evaluation:

Answers the question of how things are working. It assesses the strengths and weaknesses of a new program as it is being piloted and rolled out, providing on-going feedback to the project in order to make revisions, and fine tune the program.

Process evaluation:

Process evaluation assesses how a program worked. It examines how the program implemented the intended goals of the program. It is often considered looking into the “black box” between a plan (programs goals and objectives). It is best done when assessing the program from the perspective and experiences of all those involved in the program, from program planners to staff to program participants.

Outcome evaluation:

Outcome evaluation is used to obtain descriptive data on a project and to document short-term results. It answers what happened and the immediate effects on the targeted audience.

Impact:

Impact focuses on the long-range results of the program on the targeted audience. Impact evaluations are rarely possible because they are usually costly, involve extended commitment, and, the results often cannot be directly related to the effects of an activity or program because of other (external) influences on the target audience that occur over time.

Methodologies

Quantitative:

(Think numbers). Quantitative research focuses on the systematic gathering of numerical data and generalizing it across groups of people or to explain a particular phenomenon. Quantitative data collection methods include various forms of surveys – online surveys, paper surveys, mobile surveys and kiosk surveys, face-to-face interviews, telephone interviews, longitudinal studies, online polls, and systematic observations.

Tip: Here’s a very brief webpage and video that describes more about the topic

Qualitative:

(Think words and pictures) Qualitative research is a strategy for systematic collection, organization, and interpretation of phenomena that are difficult to measure quantitatively. Qualitative method is used to understand people's beliefs, experiences, attitudes, behavior, and interactions. Some common methods include focus groups (group discussions), individual interviews, document analysis, and participation/observations and event documentation (audio, photo, video).

Tip: Here’s a helpful source on the topic

 

Mixed-methods:

Mixed methods research involves collecting and analyzing both quantitative and qualitative data. The quantitative data includes closed-end information that undergoes statistical analysis and results in a numerical representation. Qualitative data, on the other hand, is more subjective and open-ended. It allows for the “voice” of the participants to be heard and interpretation of observations. Combining the two allows for a stronger, more valid picture of what happened and how.

Tip: Here is a helpful and detailed overview of mixed methods

Participants

  • Clients/consumers/participants

  • Artists (i.e. those delivering the programming)

  • Other informants on the “outside” of your work

    • Staff at the host site

    • Community members

    • Funders

    • Others (different contexts mean different kinds of possible sources of information – be creative!)

Data Analysis

Qualitative

Here’s a great resource from our friends in the Geoscience field – yes, Geoscience!

Quantitative

The Geoscience folks also have this helpful link to quantitative analysis