From Anecdote to Systematic: Ease on Down the Road
If you work in a non-profit organization, it can often feel like your institution runs on anecdotes.
When big decisions are made based on gut reaction, instinct, or what that rich donor said at that gala that time, you are dealing with anecdote-driven decision making.
It’s not inherently bad. Anecdotes are a form of evidence. They come from someone's experience. And often it’s pragmatic to use that information.
But when a decision really matters, when it has stakes, or when it is a big fork-in-the-road moment, putting in the work to get beyond anecdote – and getting more systematic information – is invaluable.
Anecdotal and Systematic aren't a zero-sum game. They exist as a continuum of ways to get and use evidence.
The more you can move toward strategies more toward the systematic side, the further you have moved into evaluation territory.
What’s wrong with anecdote?
An anecdote is a single story. You know, “This one time, at band camp...”
In education, those are often a stand-out moment. Or a disaster. Or that thing you swore no one would ever do… until someone did.
Those stories are true. They’re real. They matter.
But because they are one moment, they are influenced by our (faulty and biased) memories. When you start making decisions about what to do next or do differently, relying only on memorable anecdotes gets dicey.
You need data. That doesn’t mean you need some PhD dissertation. It just means you need to move toward information that you get and examine systematically.
OK, so what does systematic mean?
Merriam Webster tells me it means something that is “methodical in procedure or plan and marked by thoroughness and regularity.”
Practically, 3 qualities make something systematic:
- Procedure
- Consistent
- Focused
Procedure:
It is not willy-nilly. You’ve thought out a process in advance. There is a set of steps to follow to get the information. And to record it. And to find patterns in it.
Those steps are followed. Every. Time.
Whatever the type of information, it’s collected following a defined set of steps.
Consistent:
It doesn’t happen just when something awesome happens.
It doesn’t happen just when something awful happens.
It doesn’t happen just when your best-behaved students are coming.
It doesn’t happen when you remember to do it.
It happens on a consistent schedule. The details depend on the procedure (above) and context. But you have some plan about when to get the information the same way.
Maybe it’s every session. Maybe it’s every 3rd day. Maybe it’s Wednesdays when there’s an extra educator. Maybe it’s every day for the next week.
There are loads of ways to be consistent.
Focused:
You are not getting information about something vague. You have a purpose. And that purpose gives focus and structure to what you collect.
This matters when you are writing questions or prompts. When you're planning what to observe. When you're figuring out what to look for in student work.
Whatever your data, focus on eliciting information in the ballpark you care about. You want people responding to and revealing patterns about the same thing.
What doesn’t make it systematic? Surveys.
A data collection method does not make something systematic. The fanciest survey in the world can be used willy-nilly. Or without regard for how/when it’s distributed. Or who is responding. Or how it's analyzed.
All sources of evidence can be more or less systematic in how they are collected and/or analyzed.
Being systematic is about the HOW not the WHAT.
What else doesn’t make it systematic? Numbers.
I have not used the word “sample size” anywhere. (One of those Hunt for Red October submarine alarms goes off in my head almost any time I hear those words.)
Because my basic point is: 1 > 0
- Get started: Any systematic data is better than no systematic data.
- Keep going: More systematic data is better than less systematic data.
- Stay realistic: A massive amount of systematic data is, at best, marginally better than a lot of systematic data.
There is no magic “sample size.” If you are working in education with limited resources, most of the time you just trying to get enough systematic data to help you figure out the next right step. So, basically:
10 > 5 > 1 >> 0
But in a lot of cases:
100 = 50
And once you start collecting and reflecting on data systematically – and see how useful it is – you catch the bug. In my experience, you (and your team) will want more and more and more.
How to get started?
Your anecdotes are a great place to start! Your instincts pay attention to them for a reason.
Use them as the starting point to take the next step toward being systematic. Ask yourself: “How could we capture information about [anecdote] just a little more systematically?”
Think about a procedure that could be followed consistently with enough focus to elicit evidence of that anecdote you want to explore.
Give it a try! Ease on down the road one more step from anecdote to systematic.
It doesn’t have to be perfect to be useful. Remember: 1 > 0
Real World Example:
When we coached some museums to find practical strategies to evaluate exhibits, we found some very small institutions that just couldn't spare staff for interviews or observing visitors.
Rather than focus on what they couldn’t do, we looked at their assets.
They all had floor staff who routinely interacted with visitors in the new exhibits. These staff see a lot about how families interact and learn during their daily jobs.
But how could they capture that information in a way that would give them systematic staff reflections, rather than random anecdotes?
Procedure: Create a system around when, how, and in what form floor staff should make reflections.
Focused: Use a structured reflection form. Prompts should focus reflections on specific outcomes. Not “What did you see today?” But “What ways did families talk about engineering? Give examples you heard or discussed today.”
Consistent: Set an expected schedule of when to record reflections. Say, at the end of every shift, for the next month.*
*Pro Tip: Don’t keep collecting data unless you have the time to continue analyzing and using it
Train: Systematic processes require training. It’s important to make sure the folks who collect or provide the data know how, what, and when – so that it’s used as consistently as possible.
So, what's your institution's most legendary anecdote? You know, the one that has low-key driven countless strategic decisions for the last decade? (We all have them.)