Program and Evaluation Planning Light: Planning in the Real World

Although there are many high-quality models for program and evaluation planning, these models are often too intensive to be used in situations when time and resources are scarce. Additionally, there is little added value in using an elaborate and expensive program and evaluation planning procedure when programs are small or are planned to be short-lived. To meet the need for simplified models for program and evaluation planning, we describe a model that includes only what we consider to be the most essential outcomes-based program and evaluation planning steps: (a) how to create a logic model that shows how the program is causally expected to lead to outcomes, (b) how to use the logic model to identify the goals and objectives that the program is responsible for; (c) how to formulate measures, baselines, and targets from the goals and objectives; and (d) how to construct program activities that align with program targets.

In the ideal professional world, program planners and evaluators in higher education would have unlimited time and resources to design high quality programs, to make evaluation plans, and to get projects generously funded.However, for most program planners and evaluators the reality of the professional situation is that there is a continual struggle against ever-impending deadlines and limited resources.
There are many high quality models for program and evaluation planning, yet it has been found that in many cases they are too heavy, too complicated, too time-consuming, or require too many resources (Randolph et al., 2005).This is especially true for small, short programs that are intended to be carried out by only a few practitioners.
The model described in this article addresses the need for practical and realistic program and evaluation planning models in higher education programs.In fact, the motivation for writing this article was a result of criticism received about the feasibility of a more complicated educational evaluation and planning model for a teacher-training program in technology education, which the authors of this paper had reported in Randolph et al., (2005).
This article is not intended for planning and evaluation professionals since the information presented here is far from original.Rather, this article is intended to bridge the research-to-practice gap for practitioners who have been given the task of planning or evaluating programs.This model is appropriate when time is of the essence or when the program is so small that full-scale program and evaluation planning is inappropriate.
Examples from this article are based on a project proposal for using ICT as a tool in HIV/AIDS prevention.(See Duveskog, Sutinen, Vesisenaho and Gasso, 2003) for background on the prevention project.)Although the authors of this article put this model in the context of university level program for HIV/AIDS prevention, there is no reason that it would not work in other contexts as well.The model is simple; if it is followed correctly, it should stay within the realm of appropriate practice in program and evaluation planning.The evaluation portion of the model is especially appropriate for summative, goal-oriented evaluation (e.g., Tyler, 1949) rather than evaluation for program improvement or organizational learning (e.g., Preskill and Torres, 1999).Finally, program activities can be planned that so that they cause the targets to be met.If everything works out as planned, (e.g., if the logic model is correct, if measures are valid and reliable and align with objectives and activities, and if activities cause targets to be met), the Planning Light model should enable successful programs and evaluations to be designed when time and resources are scarce.

The Logic Model
Weiss (1998, p. 57) defines a logic model (she calls it a program theory) as "the mechanisms that mediate between the delivery (and receipt) of the program and the emergence of the outcomes of interest."Simply put, a logic model is an illustration of the causal links between the program and the outcomes of the program.It consists of mediating mechanisms, moderating mechanisms, and links between mechanisms.A mediating mechanism is simply a step in a causal chain.A moderating mechanism is one that affects the links between one or more mechanisms.A moderating mechanism can stop or affect the progression of the causal chain.
To make an analogy, in a chain of falling dominoes the program would be the first domino, the overall program goal would be the last.The dominoes in between the first and last would be the mediating mechanisms.The block or barricades that interrupt the chain of falling dominoes would be the moderating mechanisms.
Creating a logic model is useful for a number of reasons.First, it helps program designers, evaluators, and grant writers understand the program and codify their perceptions about the nature of the program.Secondly, it serves as a framework for the rest of the program and evaluation planning process.It helpful when making a logic model to brainstorm all of the stakeholders and systematically answer each of the following questions for each stakeholder group: • What are the needs of this stakeholder group?
• What are the benefits and risks that this program or evaluation will have for this group?• How should this stakeholder group be included in the planning of this program of evaluation?Considering the stakeholder questions will add depth and focus not only to the logical model, but to the rest of the planning process as well.Additional resources for constructing a logic model can be found in CDC Evaluation Working Group (n.d.); Julian, Jones, and Deyo (1995); McLaughlin and Jordan (2004); Program Development and Evaluation, University of Wisconsin -Extension (n.d.); and W.K. Kellog Foundation (2001).

Goals and Objectives
After the logic model has been constructed, defining the program's objectives and goals becomes rather easy.Although objectives and goals are synonyms, in this paper we will refer to objectives as the mechanisms that are within the scope of the program's activities and goals as the mechanisms outside the scope of the program's activities.The mediating and moderating mechanisms that fall under the scope of the program in the logic model are the program objectives.The last step in the logic model is the overall goal or goals that should describe the benefits to the program beneficiaries.The mediating mechanisms outside of the scope of the program might be referred to as intermediate goals.
It is important to note that the program will usually not be accountable for all of the mediating and moderating mechanisms in the logic model.The decision about which mechanisms to include as program objectives is usually based on several factors.The first factor depends on where in the logic model the causal chain will continue without program intervention, either because the links are firmly established or they logically follow.The second factor depends on the amount of resources available to the program.Referring back to Fig. 2, the program designers decided that the program needs only to be accountable for increasing the knowledge of intervention strategies and ensuring the accessibility and utilization of the program.It was decided that if these objectives (knowledge, accessibility, and utilization) were accomplished, the rest of the mechanisms (HIV/AIDS-related behavior change, reduction of new HIV/AIDS-cases) would occur without further program intervention and contribute to the overall goal of global health.Examples, then, of the objectives derived from the program theory of the HIV/AIDS prevention program presented in Fig. 1 are listed below: • Objectives: • create an ICT-education-based prevention program that increases knowledge of prevention strategies; • ensure that the program is accessible; • ensure that the program is utilized.
If the logic model has been correctly thought out, the overall goal should be accomplished if the objectives are accomplished.

Defining Measures, Baselines, and Targets
After the goals and objectives have been made clear, measures, baselines, and targets should be derived for the objectives (and if appropriate for the overall goal, and/or intermediate goals.)See Table 1 for an example of the measurements, baselines, and targets for each objective in the HIV/AIDS prevention example.

Measures
For each objective, one or more valid and reliable instruments need to be adopted or constructed that can measure the causal impact of each mechanism on the next mechanism or on a link between mechanisms.For example, to measure the link between the program and increased AIDS knowledge, the program designers decided to construct and validate a test of a student's knowledge of HIV/AIDS.To measure accessibility, the designers decided to measure the number of schools able to participate in the program.To measure accessibility, the designers decided to measure the number of schools that were able to participate that decided to participate.The overall goal and intermediate goals were not intended to be measured in the HIV/AIDS prevention example since the scope of the program's accountability ended at increasing knowledge.However, in other cases, it may be appropriate to measure overall and intermediate goals if resources are adequate.

Baselines and Targets
In order to answer the question, "What was the impact of the program?" one has to answer two questions

Activities
Finally, after measures, baselines, and targets have been formulated, it is now appropriate to design program activities that are aligned with the measures.Without examining how program activities lead to meeting targets, it is plausible that some influence other than the program led to the target being met.The activities should directly lead to the objectives meeting their target, as indicated by the measures.

Conclusion
This article presented a practical, outcomes-based model for program and evaluation planning that could be useful for practitioners in a variety of higher education contexts.Its strength is its feasibility; it cuts the evaluation and planning process down to the most essential steps.The steps included (a) creating a logic model; (b) identifying goals and objectives; (c) formulating measures, baselines, and targets; and (d) aligning program activities with targets.The model's weakness is that the emphasis on evaluation feasibility comes at the price of a de-emphasis on evaluation utility, accuracy, and propriety.Furthermore, the model works best in rather static, foreseeable environments, which makes it weak in programs with unknown or undecided outcomes.Overall, given that every precaution is taken to reduce harm to stakeholders, if there is only time or resources for a few evaluation and program planning activities, we suggest that these are the activities that be done.
The following sections summarize the Program and Evaluation Planning Light model (hereafter [the Planning Light model].)It consists of four systematic steps: 1. Creating a logic model.2. Identifying goals and objectives.3. Formulating measures, baselines, and targets.4. Aligning program activities with targets.Fig. 1 illustrates the logic of the Planning Light model.Creating a logic model clarifies the nature of the program and enables goals and objectives to be identified.After goals and objectives have been identified, measures, baselines, and targets can be formulated.
Fig. 2 illustrates a logic model for an ICT-related HIV/AIDS prevention program.This figure illustrates how the prevention program is intended to bring about knowledge of prevention strategies, HIV/AIDS-related behavioral change, reduction of new
-"What was, or would have been, the state of the objective before, or without the program?" and "What is the state of the objective after, or with, the program?"A baseline measurement indicates the state of the objective before the program existed or predicts the state of the program had the program not existed.A target measurement indicates the intended state of the objective after, or during, the program.Pre-tests and control groups measurements are types of baseline measurements.Post-tests and experimental group measurements are types of target measurements.See Table1for an example of baselines and targets for the HIV/AIDS prevention program.