I love it when a plan comes together!
It's the unsexiest part of evaluation work. It's the part that involves meetings and note-taking and white-boarding. It's the part makes you impatiently ask, "Um but when are we going to get some data?!?"
And it's the part that routinely gets short-changed in timelines.
But if you want a cost-effective evaluation: Plan slow. Implement fast.
That AMNH project I mentioned? We spent more time planning the evaluation than we did implementing the study.
It wasn’t sexy. It wasn’t straightforward. It wasn’t simple.
But, in hindsight, I see about 15 project landmines that we avoided by not rushing in. When you have to rush to collect data as quickly as possible, it's a gamble.
I often hear timeline assumptions that leave a couple of weeks to plan a study. I get it! You are eager to get that sweet, sweet visitor data in your hands.
But evaluation is a big investment.
Investing more time in planning is a secret to getting the absolute most bang-for-your-buck from the study.
I'm going to share one concrete process you can use to help your team invest time in planning that will maximize the value of your evaluation -- whether it's in-house or out-house.
Stage 1: Lay all the cards on the table.
You can tackle this before calling in the evaluator. Get key stakeholders together and get everyone to articulate their evaluation wish-list:
- What are your hopes for evaluation?
- What are you wondering about that you think evaluation could help you know?
- What claims or promises have we made about what this will do or achieve?
Get everyone to unpack their evaluation baggage now. Because if a rogue, "Wouldn't it be great if we could..." pops up the week of data collection, it’s too late.
It is likely to be… a lot.
That's good! Now you (and your evaluator) will know the field of play. No more reading the tea leaves.
At the same time, it is important that people don’t leave this session thinking an evaluator can make it all happen (affordably). So, close the brainstorming with something like:
We've generated a lot of great ideas! Realistically, we won't be able to evaluate all of this. But this gives us clarity to narrow in on the pieces that are most important and most doable.
Stage 2: Bring order to the chaos.
Now’s a good time to bring in an evaluator assist. Some Dos and Don'ts.
Don't hand this list to your evaluator and say, "We want to evaluate all of it. Go." That makes us sad.
Do share this as context of where you are at and ask for advice. At this stage, an evaluator can often group ideas in ways you might not see. We can see overlapping questions and directions that bring some order to the chaos.
By grouping random questions into coherent lines of inquiry, I can give you a smaller set of questions to think about.
Stage 3: Prioritize
Now it's time to make some tough choices.
You will almost always pose more questions than we can feasibly answer with a single study. You're curious! And that's great.
But to help you narrow the field, here are a few questions to ask your team:
- What is most important to answer with evaluation right now?
- What are we under pressure to answer?
- What will allow us to make our next decision?
- What sounds interesting... but isn't likely to influence what we do?
Alongside this prioritizing, ask your evaluator for a sense of which questions are a bigger or lighter lift. Some may be more feasible to answer than others -- which can sometimes tip the prioritization scales.
This is why investing in planning really matters.
There is no universally right answer to these questions. When your institution identifies what really matters, it will drive a study that gets information you can actually use.
Stage 4: Plan for flexibility
Especially if a project is big, complex, or messy, leave a little wiggle room in the final plan if at all possible.
When projects finally become "a real boy," there's often a last-minute or mid-project de-prioritizing and re-prioritizing process that happens.
Leave time for this. There will come a point of no return. But if you can leave a little space for key moments to make adjustments before that time, it can often make the most of your planning.
What to do when the calendar is your enemy?
Maybe you are totally bought into the idea of investing in planning. But you end up in a very common scenario where the green-light for evaluation is made late in the game. What do you do?
When push comes to shove, I jump to Stage 3. Generate questions directly from the lens of prioritization. You still want to articulate your expectations for evaluation. But leave a less space for blue-sky dreaming. Ideate and prioritize at the same time.
Then, trust your evaluator for advice on how feasibility balances with priorities. And it's really and truly OK not to evaluate everything.
Real World Example:
In the AMNH project I mentioned, we used these stages with Karen and her team to arrive at a solid plan for a massive and complex space. Or at least for a starting point.
Phase 1: Cards on the Table
- Karen gathered the full spectrum of questions from internal stakeholders. There were 40+ unique questions posed.
*PSA: If you have 40 unique evaluation questions, you do not have a study.
You have a brainstorm.
Phase 2: Order to Chaos
“How do you eat an elephant? One bite at a time.”
First we organized the questions into a Timeline of Relevance:
- Wave1: Answers we need NOW.
- Wave 2: Answers that would be easier, if we had some Wave 1 answers first.
- Wave 3: Answers that require long-range research or time-scales.
- Then we grouped the remaining NOW questions into a few distinct lines of inquiry.
Phase 3: Prioritize
- We ruled out questions that internal staff or data could answer easily – or well enough. (Perfect is the enemy of the good when prioritizing.)
- We found the questions that were most pressing and could drive decisions.
Phase 4: We Pivoted
- Right after the space opened, we realized one of our lines of inquiry wasn't as big of a mystery as they thought.
- We axed it. Then we plugged in a component we’d had to cut.
7 months of planning, scoping, and revising.
2 months to get the first wave of results in staff's hands.
Plan slow. Implement fast.
What have been your experiences with planning - fast or slow? Roadblocks in either route? Reply and let me know.