Hi Reader,

If you met me years ago, you probably know me as a runner. But the past six months have had me completely sidelined due to an injury that, to everyone's surprise, was 0% caused by running.

One of my rehab companions has been the Peloton App. If you’ve been exposed to Peloton – even via those silly commercials – you may know that each instructor has their catchphrases. Something they say to start or sustain motivation.

Angie and I occasionally compare notes on favorite instructors. And one such conversation led to this month’s topic:

Angie noted that we are Denis Morton Evaluators.

What does that mean? Well, it has to do with who's the boss* in an evaluation project.

(*Mona was the boss. No question.)

Also included is:

• Workshop Opportunity at ASTC in September
• Celebrating Michelle & the Visitor Studies Association

Cheers,

Jessica

PS – I feel obliged to say that we are in no way sponsored by or affiliated with Peloton. But hey, if you know someone, tell them to call me.

We make suggestions. You make decisions.

If you also succumbed to the Peloton App as a route to at-home workouts during the pandemic, perhaps you encountered this mantra, from instructor Denis Morton:

In the context of a workout, this is an acknowledgement that health should not be a forced march. Your instructor is an expert. But you are the expert on your body.

This statement comprises the entire sliver of the Venn diagram of what our Evaluation Team has in common with a Peloton instructor.

I really love this statement because it encapsulates how I view our job.

We make suggestions. You make decisions. And we want you to be empowered to make the decisions that are right for your organization at this moment in time.

What does that look like in reality? Let's dig in.

Study Recommendations

This can be a fraught part of any evaluation report.

Be honest: Have you ever received a report from any consultant that contained a list of recommendations, skimmed down the list, and had any of the following reactions?

• Um, we already do that.
• We did that 10 years ago, and here’s why we don't anymore.
• And how do we do that without an extra $100k?
• Wait, we’ve been saying that for 5 years. So they listen to you? But not us? Cool.
• And how do they propose we get around [insert barrier here]?
• What would that even look like?

These aren’t inherently a defensive knee-jerk reaction. (I mean, they can be. We're all human.) But often, they're a reaction to recommendations that feel written AT you, instead of WITH you.

This is why we rarely include a “Recommendations” section in our reports - unless they were co-constructed with our clients. Say, at a Data Party.

We don't just throw data at you and shrug. There are always Conclusions to tie the biggest ideas together with a bow. And often there are Implications to connect the dots with your program design.

But these are, at most, suggestions.

A written list of “Here’s what you should do next”? Nope. I have never, not once, in my 20-odd year career felt comfortable straight-up telling an educator: "Do _____."

There’s never a single solution to a problem. Or way to build upon a success.

The data should open eyes and lead to change - and maybe a little discomfort along the way. But it can't do that in a vacuum. When it comes to turning data into decisions, the expertise of staff needs to be directly in the mix.

We can connect the dots and suggest a direction.

You decide where to go next.

Study Design

When you start working with an evaluator, I think the expectation is that we tell you what methods to use. We’re the experts, right?

In that case, I guess I can be a bit frustrating.

Because I’m infamous for being in conversation about a project and promptly rattling off 3 totally different methods that could address the need. But we can't do all of them.

I just can't think of evaluation in terms of right and wrong. Every method has strengths. Every method has limitations. Deciding is a matter of weighing those against each other.

And you, as someone in charge of an educational experience, might weigh those factors differently than I do.

We often introduce a Methods Menu during planning, in order to outline several different methods that could work, alongside the strengths and limitations of each, in terms of:

• Types of data
• Amount of data
• Difficulty of getting that data

For me, this is a way of presenting options for study design, with our expert take on the pros and the cons.

We can suggest what we favor in that mix of options and why.

But ultimately, you need to decide how those factors weigh for you to make a decision about what’s right for your project and what you want to say about it, at the end of the day.

Data Analysis

Psych! OK, this is the area where our roles are reversed. At this phase, you may make suggestions, but we make decisions.

Analysis is where things get really technical. We are constrained by earlier decisions that determined what type of data we have. (This is why the planning stage matters so much.) We can’t magic statistics worth having into existence.

But with that said, your suggestions at this phase are still important. Stuff can change between planning stage and analysis stage. When we hear from you about analysis, it can focus where and how we look within our data.

Let’s say, at the last minute, your funder suddenly expressed that they really care about institutional change – even though they shrugged it off before. Is there any chance the analysis can address that? Or, hey, we've been wondering if there is any difference between families and adults? Or, is there a statistical test that could prove X to get VP Muckity-Muck off our back?

At a technical level, we have to make the call about whether a particular line of inquiry is right for the data we have. But within the those bounds of reality, your interests help us make the final analysis speak to you. (And your funders.)

In this case, you make the suggestions.

We make the decisions right for our data today.
​

Real World Example:

We've facilitated a lot of really fun Data Parties over the years. But there is one that sticks out in my memory.

This group had tackled some thorny, complex, and VERY qualitative data over the past 2 hours. I mean, I doubt there was a single number in the data that day. It was all themes and nuance and themes within nuance.

Nevertheless, this group really BYOE-ed! (BYOE = Bring Your Own Energy —Callie)

After Michelle and I packed up our toys (chart paper, markers, post-its, and data placemats), we noticed something weird.

Our session participants... hadn’t left the room.

At first, we assumed they were shooting the breeze, before heading back to desks scattered around campus. But after 10 minutes, that was pretty evidently not the case.

We sidled up a little closer and realized: It was an impromptu staff meeting. They were game-planning a bunch of specific next steps they needed to take based on of this data discussion. And, like, now.

We were floored.

Usually, a successful session gets us to some concrete suggestions on a piece of chart paper that we will synthesize in a document.

But this time, our data made some suggestions. And, by golly, those staffers were MAKING DECISIONS.

We hung back and just smiled. The torch had been passed. We completed our part. Now they were doing theirs.

Does this mindset work for you? Or does it freak you out to realize you can have this much power over the direction of your project’s evaluation?

​
PS – Would it be weird if I confessed to starting big a big burst of report writing by shouting "Yes Yo!" in my empty office? It would be weird, wouldn't it? Yeah, OK, I've never done that. Never.

Workshop Opportunity at ASTC

Speaking of making decisions about where to go next — want to make some of your own about an evaluation plan you've been avoiding?

As a pre-conference workshop at ASTC in Phoenix 😎 in September, I will lead: Useful and Feasible: A Realistic Approach to Evaluation Planning.

September 16, from 2-6 p.m.

Arrive a little early and spend the afternoon learning (in that sweet, sweet conference center air conditioning).

This approach is entirely practical. Entirely customized to you. I'll offer frameworks, structures, and pro/con lists. Each step of the way, you'll make the decisions to plan an evaluation that works for your resources and capacity.

A past participant told us:

Not sure if it's a good fit? Hit reply and ask me.

🥳Congratulations, Michelle!

We want to celebrate our team member Michelle Lentzner. (We want to do that every day, but this month is special.)

Congratulations on four years of board service to VSA, as the Conference Committee Chair!

I know, from ancient personal experience, how tough that specific job is. And Michelle has done a great job. We’re so proud and impressed!

Reader, will you be at VSA’s Annual Conference next week? If you see Michelle there, tell her congratulations and thank you! (You might embarrass the heck out of her, but I'm OK with that. 😊)

Or, better yet, stop by one of her fantastic sessions on the program:

• Ripple Effect Mapping: The program ended… and then what happened? (Tuesday, 7/21 at 2:45)
• Hills to Die On: The Visitor Studies Showdown (Thursday, 7/23 at 11:45)

Should be a blast!

P.S. Know someone who'd benefit from these ideas? Forward it along! (Sharing is caring.)

P.P.S. Get this from a colleague? Sign up to get practical insights like these every month.

Why the "Evaluation Therapy" Newsletter?

The moniker is light-hearted. But the origin is real. I have often seen moments when evaluation causes low-key anxiety and dread, even among evaluation enthusiasts. Maybe it feels like a black-box process sent to judge your work. Maybe it’s worry that the thing to be evaluated is complicated, not going to plan, or politically fraught. Maybe pressures abound for a "significant" study. Maybe evaluation gets tossed in your "other duties as assigned" with no support. And so much more.

Evaluation can be energizing! But the reality of the process, methods, and results means it can also feel messy, risky, or overwhelming.

I've found that straightforward conversation about the realities of evaluation and practical solutions can do wonders. Let's demystify the jargon, dial down the pressure, reveal (and get past) barriers, and ultimately create a spirit of learning (not judging) through data.

This newsletter is one resource for frank talk and learning together, one step at a time.

​Learn more about JSC and our team of evaluators. Or connect with us on LinkedIn:

​