PROCESS
The PEP-AH Process
The first step in the PEP-AH process, engaging a dedicated team, is covered elsewhere in this guide. There are four remaining stages in the process and each section below covers one key stage. These stages do not necessarily progress in a linear fashion. They are:
- Engaging a dedicated team
- Assessing what is already happening on your campus
- Planning an initiative
- Implementing an initiative
- Evaluating an initiative
ASSESSMENT
We recommend that campus teams do some upfront work to understand the place of alcohol on your campus, think through what protective strategies are currently working and assess what’s already happening to address alcohol harms. Your team might know about what is and has happened on your campus or might need to do a little digging to find out. This information can help guide your work in planning, implementing and evaluating initiatives as part of the PEP-AH framework.
What does alcohol consumption look like on your campus? Every campus is unique; alcohol consumption and harms can be quite different from campus to campus. Campuses vary by population, size and location, and academic and social calendar. All of these features affect alcohol harms. Gather information about alcohol use on your campus using the PEP-AH Campus Alcohol Culture Worksheet. When you understand your campus, you’ll be able to better address its profile of alcohol harms and improve the relationship of students to alcohol on your campus.
How do students reduce their exposure to alcohol harms on your campus? When students on your campus are choosing not to drink alcohol, where do they go and what do they do? What healthy, fun activities and initiatives could you promote or encourage? The protective strategies that your campus uses can help to build a culture of moderation that can help reduce alcohol harms. See the PEP-AH Campus Alcohol Culture Worksheet for more details.
What specific work is happening on your campus to address alcohol? We’ve developed a spreadsheet where you can document that work across different strategic areas ( PEP-AH Self Assessment Resource .xlsx file, 3MB). Your team can break up the work and gather this information from key people on your campus. We find that it is helpful to have one person responsible for pulling all of the information together, and other team members tagged with reaching out to contacts on your campus to gather information on strategic areas.
Your campus might be small enough that to review existing work on alcohol harm reduction takes less than one meeting because one member of the team knows everything that is happening. However, on some medium or large campuses the review could take longer. Take the time to identify the initiatives on your campus, as the information will be valuable in planning your own initiatives.
Once your team has a profile of what’s happening on your campus, offer to keep staff or volunteers involved in other initiatives up to date on the activities of the PEP-AH Campus Team and invite them to keep the team informed about their activities. Connecting might mean regular communication by e-mail, sending out your meeting minutes or having someone from your team attend another meeting where alcohol harm initiatives are addressed. This communication can help identify important areas to build initiatives by getting a range of perspectives on alcohol harms. Coordination can help ensure that your PEP-AH Campus Team isn’t brainstorming new initiatives that are already underway elsewhere on campus. We encourage you to communicate and coordinate as much as possible.
Once you know what initiatives are already happening on your campus and you are communicating with the right contacts, review the PEP-AH framework. It could be fairly easy to identify next steps based on an analysis of what is missing on your campus. Some actions might be relatively accessible and quick; some strategic areas are longer-term commitments. Whatever is of most value to your campus, we encourage you to jump in and get started.
- Talk to campus services (physical plant services, health services, food services, library, security). When a call comes in about alcohol, security services are often responsible for a response. We have found that these units are a wealth of knowledge that can help your team understand the specifics of problematic alcohol use on your campus.
- Talk to students who drink alcohol moderately or choose not to drink. In shifting your culture towards moderation, students who already embody the ideal of moderation can help you to understand alternatives to drinking on your campus. What other activities are open to students on common partying nights? In directing students who choose to drink away from alcohol, what exists that can replace drinking?
- Revisit your assessment. As you begin your work as a PEP-AH Campus Team, it is valuable to review all of the strategic areas at one time. You might discover good harm reduction initiatives in place. Note the good work and record it for future reference. Revisit your assessment once a year to make sure it is up to date and that no new initiatives have been started by others on your campus in the time since your last assessment.
Planning
This section will help your team to plan initiatives based on the PEP-AH framework. This is the time to look at your priorities and develop an implementation plan. This section will help you to address your priorities and consider the evidence, context and facilitation approach needed for a new intervention. You’ll then be ready to map out the initiative using a logic model and set up indicators of success.
We suggest that you look at your campus culture, what’s already happening on your campus and set your priorities based on what your team thinks will have the most impact. The initiative needs to be achievable with current resources and within set timelines. Don’t spend too much time debating the perfect priority area: pick one initiative and get started. The PEP-AH framework represents our best attempt to pull together effective initiatives in one place. Refer to it to guide your planning.
As you plan the implementation of your identified initiative, consider a few ideas from the PARiHS framework (National Collaborating Centre for Methods and Tools, 2017):
Evidence: What is the nature of the evidence for this initiative that led your team to choose it over others?
Context: Where will your team be putting this initiative into place?
Facilitation: How will your team facilitate implementation to ensure that you are aligning the work to suit the evidence and the context?
For more questions to guide considerations for planning and implementation, see the PEP-AH Planning Worksheet.
A logic model is a simple way to describe your initiative. The logic model template below can help you identify needs of the initiative (Resources), what will happen as part of the initiative (Activities), how you would know the initiative happened (Proof of Completion, sometimes called outputs; for example, people attend an event or social media posts are posted), and what you expect to see as a result of your initiative (Outcomes). You can also differentiate between short- and medium-term targeted changes (Short- and Medium-Term Outcomes) and the long-term impact (Long-Term Outcome).
We recommend that someone on your team lead the development of a draft logic model. The team can then work together to finalize it. While it might be tempting to include a lot of detail, the best logic model is an easily understood snapshot of your initiative. For example, if your initiative included handing out water bottles at every entrance to a sports event, your team might be tempted to add the activities, “Stock each table with water bottles” and “Hand out water bottles at every entrance” as activities. If there are a number of other activities to list in your logic model it might become too big to be useful at a glance. Consider labels for activities such as “Distribute water bottles” and explain the exact details elsewhere, if necessary.
Building a logic model also provides the team with an opportunity to think about how you would know you were successful. The PEP-AH Outcomes and Indicators Resource in the PEP-AH Action Guide Worksheets can support you in determining how you might measure your initiative with one or two key indicators.
Be sure to keep your logic model up to date as you continue to plan and implement the initiative (e.g., through improvement cycles, see below).
Indicators are simply measures of your outcomes. You must be clear in the planning phase about what you are doing and what outcomes you plan to achieve. For instance, if an outcome is “reduced clean-up costs following events,” a valid indicator might be the number of calls to site services to clean up during and following events on your campus. If these calls have been tracked at past events, your team will be able to track changes over time. You might have key indicators that you want to begin to track with the current initiative. We suggest a limit of three to five indicators, as tracking too much information can be time consuming and take away from implementing initiatives. As noted in the PEP-AH Outcomes and Indicators Resource, seek out indicators that are the most meaningful, useful and realistic.
- Set a date. If you set a concrete date for your initiative it is easier to get started. Your initiative doesn’t need to be perfect to be effective. Get started and your team will gain both confidence and experience. In the section on Implementing an Initiative, you’ll find resources for adjusting your approach as you go.
- Plan together. It is important that your team gets together or is kept in the loop during the planning process. Planning the details of initiatives can change them slightly, where your initial idea about resources or activities might shift slightly. Some of these changes might be important to your Campus Team members, and without their input you won’t be able to bring their perspective to your planning.
- End your meeting with next steps. Some of the best conversations with your Campus Team will be wide ranging and could include a number of potential areas to follow up. Take the last five to fifteen minutes of your meeting to state all of the next steps for follow up and completion. Everyone must be clear on what needs to be done, by whom and by what date. We suggest that these next steps go out to the team via e-mail. That way the whole team can keep track of what will happen next and who is responsible to make it happen.
IMPLEMENTATION
This section will guide the implementation of your initiative. You have the right team, you have a plan and now you are ready to make a difference on your campus. Consider how to implement your initiative based on the available evidence, the context and good facilitation. Get ready to monitor your indicators of success, and fine-tune your initiative as you move forward.
In planning your initiative, you used a logic model to lay out reasonable outcomes that flowed from the activities. Your team then considered the most important and feasible indicators of success. Now it is time to consider how you can monitor those indicators. This monitoring will look very different across different initiatives.
We have found it helpful to make one team member responsible for seeking out the information that makes up your indicators and keeping that information organized and accessible to the team. Information could include security or ambulance calls, or measures like surveys or feedback forms.
Be ready to identify opportunities to adjust and make improvements as implementation gets underway. You can use the widely accepted Model for Improvement to enhance the effectiveness of your initiative. It is based on three essential questions and the Plan-Do-Study-Act cycle (Langley, Moen, Nolan, Nolan, Norman, & Provost, 2009). Ask the following questions:
- What are we trying to accomplish?
- How will we know that a change is an improvement?
- What change can we make that will result in improvement?
These questions feed into the Plan-Do-Study-Act cycle of testing an improvement.
For example, your Campus Team has an initiative to distribute water for students to drink between alcoholic drinks at campus events where alcohol is available. While implementing the initiative at events, your team notices that some planned activities are not leading to the targeted changes you were hoping to achieve. You might notice, for instance, that fewer students than expected are taking the water that is offered.
Your Campus Team comes together to develop a plan to address the issue (e.g., change the water distribution location so that it is closer to the alcohol service). At the next event, you do what you planned and note any changes. At the Campus Team meeting following the event, set aside time to study the results (e.g., note any change in number of water bottles distributed, observations of the people handing out water and incidents of harmful drinking at the event). Once your team has agreed on the interpretation of the change, you act by changing the plan to include the improvement (e.g., the team decides to always distribute water near the alcohol service).
The Plan-Do-Study-Act supports constant improvement, meaning your team can continue to cycle through the questions and phases. The essential ingredient is fine-tuning, adjusting your approach gradually so that you can understand what is contributing to your successes and what is not, and then continuing to do what works. We suggest using the Plan-Do-Study-Act Worksheet (included in the PEP-AH Action Guide Worksheets) as a team to plan how you’ll fine tune your initiative and understand what adjustments were successful. Changes that you make to the initiative through improvement cycles can be noted in the logic model.
- Adjust your initiative as needed. With every initiative, there will always be unforeseen opportunities and challenges. Don’t be discouraged! Adjust the work to continue to move towards achieving your planned outcomes.
- Make small adjustments one at a time, where appropriate. Sometimes we see a number of areas for potential improvement. It might be appropriate to make several adjustments at once if the initiative happens once per year, like an awareness event. But if the event happens regularly, such as non-alcoholic events, your team might want to make one change per event so that you can understand which changes are contributing to your success and which changes are not.
- Balance staying true to source material and adapting to your context. There are initiatives that lend themselves to exact replication, and there are interventions that need to be adapted. As your team implements initiatives, work towards balancing what you change to suit your context and what you maintain to be true to a model initiative. The more you adapt, the more you need to think through the elements you have added or taken away. How will they affect the targeted changes you are hoping to achieve? How will that change what you need for the initiative and the activities that you will do?
Evaluation
This section discusses the evaluation of the initiatives that you’ve planned and implemented. It includes guidance on how to appoint a team member to lead evaluations, set evaluation questions, collect and review data, conduct your evaluation based on the best available information, and then use your evaluation to inform next steps for that initiative. Finally, this section includes some practical steps your Campus Team can take towards evaluating all of your campus work.
While planning your initiative, your team identified target changes (outcomes), as well as ways to measure change through key indicators. Your team can evaluate your initiative by using your indicators to answer key questions about your initiative. This might mean determining if you have achieved a desired target for an important indicator or looking at the differences between indicators over time. There are many different ways to evaluate outcomes and answer evaluation questions. Your team might have the research methods and program evaluation capacity to perform in-depth analyses on a number of different measures. However, you can also do simple analyses and arrive at valuable insights about initiatives. These insights can help your team understand how to improve initiatives and the overall value of initiatives.
If one member of your Campus Team has capacity in working with data and indicators, and interpreting data, they would be a natural fit to lead evaluation efforts. This role would pull together information about the initiative’s indicators, lead analysis of the information, present results and the interpretation of the data to the campus team, and help to determine next steps. If the team has decided that it’s best to bring in outside help with program evaluation, it would still make sense to have a team-based lead to facilitate the evaluation. See the resources below for more detail.
Evaluation questions are the main questions you want to answer about an initiative through evaluation. Some questions will relate to the resources and implementation (processes) of your initiative and others to the outcomes. As a rule, process questions are appropriate earlier in the life of an initiative and outcome questions are appropriate for long-standing programs. We have found that a mix of different process and outcome questions are usually appropriate for alcohol-related initiatives.
Process Evaluation Questions
- Resource Questions
- Did the program have the resources that it needed?
- How did those resources affect implementation of the program activities?
- Activity Implementation Questions
- Were activities executed as planned?
- What activities contributed to the success of the initiative?
- What were the barriers and facilitators for implementing the activities of the initiative?
Outcome and Impact Evaluation Questions
- Outcome Questions
- Did the program lead to the planned short- and medium-term outcomes?
- Did the activities specifically lead to the planned short- and medium-term outcomes?
- Were there any unexpected positive or negative outcomes?
- Were the indicators adequate measures of the outcomes?
- Impact Questions
- How did the initiative ultimately impact the targeted audience?
- Was the impact of the initiative maintained over time?
- What level of impact can be attributed to the initiative?
After determining the questions, campus teams need to identify your information sources so that you can adequately answer the questions.
Collect and Review Data
When it is time to collect and review data, it is helpful to have a team lead assigned to this task. Once the needed information on the initiative is in the same place, it is worthwhile to quickly examine it. If, for instance, your group used a standard event feedback form to assess the overall satisfaction of attendees, an initial review might show that very few participants completed all questions, and this gap might have an effect on your data that you need to be aware of when interpreting the findings.
Use simple analyses to help move your initiative forward unless your Campus Team has the time and capacity to break down data using more complex analyses. If your team has some capacity to run appropriate calculations, then consider analyzing the data to understand it better. This analysis might include a number of different breakdowns or tests of significance depending on the data and capacity of your team. For qualitative information, this might mean deriving themes from interviews or focus group notes if appropriate.
Interpret Data
We suggest that your evaluation lead walk the Campus Team through the information collected and compiled in relation to the evaluation questions. The evaluation lead can help the team interpret the data. While not all team members have experience or expertise in interpreting data, all team members have important perspectives to bring to the interpretation of data. Your team can document the interpretations and the answers to the evaluation questions to share with others or to refer to in your later work.
Your Campus Team is now in a position to do something with the completed evaluation. The team has likely gained some insights about the process and outcomes of the initiative, about the indicators and measures, and about how implementation can be enhanced in the future.
With these insights the team can:
- Think about its priorities and if this initiative has proven valuable enough to continue or not.
- Continue to improve the implementation and processes of the initiative through the improvement model and the Plan-Do-Study-Act process.
- Update or improve information sources to make sure that they are serving the purpose you hoped they would, as indicators of the planned outcomes.
- Make concrete plans to incorporate what you’ve learned about how your Campus Team is organized and carries out programs, initiatives and events.
- Consider documenting the findings for future reference.
Program evaluation helps to provide practical insights, improvement and judgments about programs or initiatives. Make sure that the utility of the program evaluation outweighs the time and energy it takes to complete it.
Now that your Campus Team has some experience evaluating individual initiatives, programs or events, you might want to evaluate the progress of other initiatives. Your evaluation of individual initiatives can help you gain insights about the effectiveness of your combined initiatives. Extending and linking evaluations might help to build common indicators across initiatives that can help you answer questions related to the ultimate impact of your work to reduce alcohol harms. For instance, your team has a number of initiatives across four of five strategic areas. All of the initiatives lead to increased use of protective behavioural strategies (medium-term outcome) and reduced alcohol harms (long-term impact). Your team might be able to use common evaluation questions across initiatives to understand if your combined efforts are achieving your outcomes.
- Don’t detract from the work to obtain perfect information or evaluation methodologies. Your Campus Team came together to address alcohol harms. PEP-AH partners want to encourage you to spend as much of your time and effort as possible developing and implementing programs, initiatives and events to address alcohol harms. Use the information, indicators and program evaluation resources at your disposal to do the best evaluation possible, while leaving the majority of your time to do the work to reduce alcohol harms on your campus. Implementing promising or proven initiatives is the first priority of PEP-AH. Collecting data and evaluation are important, but are meant to help enhance implementation.
- Interrogate your information. We are all subject to biases towards our own good work. We want to do good work and be a part of doing good work, and so we’re more likely to falsely confirm our work as good. Evaluating your program is the time be critical about what your information can tell you and what it can’t tell you, and about the ultimate utility of the initiatives you put into place.
- When in doubt, keep it simple. Evaluation is an important part of implementing successful initiatives, but if your Campus Team is short on evaluation resources, keep it simple and think about evaluation as just a larger Plan-Do-Study-Act cycle.
SUSTAINABILITY
This section of the toolkit is intended to help you think about how your Campus Team can sustain a comprehensive approach to address alcohol harms on your campus. We encourage you to continue to think through how you can engage a strong team around this work, assess what is happening on your campus in relation to alcohol, and plan, implement and evaluate new initiatives. The PEP-AH framework’s strategic areas include individual, environmental and system approaches. In our experience, individual approaches are more easily implemented than environmental or system approaches
It can be tempting to plan and implement an initiative without worrying too much about the future. We want to encourage you to get started, but it is also a good idea for your Campus Team to discuss how you can sustain initiatives on your campus over time, particularly in light of the fact that there will be regular turnover in the membership of your team.
In the PEP-AH Guide to Getting Started, we encourage considering extended team members. In planning your initiatives think about the following questions:
- Could your initiative be taken over by a specific, permanent group or unit on campus?
- Are there ongoing sources of funding for initiatives? Can you structure your initiative so that it could be eligible for this funding?
- If all members of your Campus Team were to leave campus suddenly, could a new Campus Team successfully repeat the initiative? What would you need to develop to help others to continue or repeat the work?
Students graduate, faculty retire and staff members get new positions. As much as possible, think about how your Campus Team can be set up so that its work continues. As mentioned above, there could be opportunities to link the work of your Campus Team to groups that can help supply members, volunteers or intern positions that would be available every year.
There is value in continuing to assess initiatives already underway on your campus in relation to alcohol harm reduction. Monitoring what is already happening on your campus can help to keep PEP-AH initiatives aligned, informed and current.
MOVING FORWARD:
Continue to Plan
As your Campus Team continues to implement initiatives, it can be valuable to revisit planning documents like the logic model or indicators for initiatives. In the process of adapting your work you might find that your initiative looks very different from when you initially planned it and there is an opportunity to rethink what outcomes your adapted initiative might achieve. We have also found value in thinking about new initiatives by starting with tools like logic models and thinking through what indicators will help us to measure success early in the planning process.
Continue to Implement
As you continue to implement initiatives, we encourage you to keep using tools such as the Plan-Do-Study-Act cycle, and to collect information that will help with your evaluations.
Continue to Evaluate
Over time, you might find that you have new evaluation questions that you want to answer. Your Campus Team might become more interested in evaluating outcomes and the impact of your initiatives. It might also be appropriate to simply continue to collect the measures related to your indicators and use them in Plan-Do-Study-Act cycles as the extent of your ongoing evaluation.
