Hi Reader,

In a recent meeting about a teacher PD program we're evaluating, I was introduced to a teacher meme that I shall call the Skibbidi Rizz Rubric.

It is perfection. The approach to translating teacher speak into slang of The Kids Today to finally get across, "Want that A? Here's how," is hilarious. Based on our work with middle school teachers, it checks out.

This month, let's dig into rubrics -- why they are flexible and powerful tools for evaluation. Including, as the Skibbidi Rizz Rubric Teacher shows, how coming up with user-centered labels for rubric levels really is an important part of the process.

Also:

• Want more rubrics? We have a resource for that.
• Where to find us in-person this November.
• Don't forget those grant deadlines...

Cheers,

Jessica

Rubrics: How do I love thee? Let me count the ways.

What comes to mind when you hear "rubric"? For many people, the last time you heard that word was a teacher communicating grading expectations.

While I never approach evaluation as giving a grade, the basic principles and structures of rubrics hold true when you apply them to evaluation. And their ability to capture nuance while quantifying depth is unmatched.

Rubrics​
How do I love thee? Let me count the ways.

I love thee to the depth and breadth and height

That you kick ass at measuring.

-with apologies to Elizabeth Barrett Browning

​

Let's bust up some jargon and talk about how this tool brings the magic. When I say rubric, I mean a system for categorizing or scoring data that has two key features:

1. Identified sequence of levels. Levels can be about performance, depth, quality, progress, whatever. But those levels are ordered. Low to high or high to low.
2. Clear definition of criteria. You clearly define what each level looks (or sounds, smells, tastes*) like in practice. Each level is concrete, descriptive, and tied to reality.

*Could you have a multi-sensory rubric for, say, craft beer? Yes, yes you could.

**Do I have such a rubric for craft beer? No, I opted for a semantic differential scale instead.

​

Why do these two qualities make rubrics magic?

1) Ordering defines a hierarchy that matters.

This is not a tool for an all-answers-are-created-equal situation. This is not to explore all the ideas that exist. Using a rubric puts a stake in the ground. You acknowledge that the thing you care about manifests on a continuum and that it is helpful to know where things fall on that continuum.

This doesn't mean rubrics have to be heavy-handed or judgmental. Take a few steps back from the "pass/fail" type language you may recall from formal education. Especially when you'll be sharing with stakeholders invested in the data, think about how they see stages of progress.

In implementation evaluations, for example, we've relied on levels that reflect the progression of learning and applying a new model, such as:

• This was tough
• Making progress
• We did it!
• Above and beyond

​

2) Defining levels makes success concrete.

This is the hard work of developing a rubric.

There is huge value in taking the time to spell out each stage of accomplishment. What does it really look like to be incredibly successful? What does it really look like when it bombs or is neglected? What does solid, most-of-the-time performance look like?

The worst rubrics I've encountered fail at this part. Bad rubrics are opaque, vague, or essentially say, "I know it when I see it." They fail to make concrete what constitutes the difference between low, medium, and high. Red flag that you have a crappy rubric? Self-referential definitions.

While you spell it all out, you also draw big, thick borders between the levels. To systematically apply the rubric to data, you need to know what kicks something up from a 2 to a 3. And the criteria need to be concrete.

You can call the borders in a rubric artificial, and I won't fight you. But you can't call them arbitrary

3) Sequence + Defining Levels = Focus

If you commit to both of those principles, it will bring tremendous focus to your evaluation measures. Each rubric can look at just one thing. If you try to define levels for two different indicators within one rubric, it can get really tangled up.

The process forces you to get really clear that you care about this One Thing and how much / well / deeply it is happening.

Care about another One Thing? It needs its own rubric.

4) There are so many potential data sources.

A rubric is an analysis tool, not a method. To use a rubric, you need a data source to which you apply that rubric.

This also makes rubrics so powerful. They can apply to lots of types of data. Here's a list of few data sources where we have applied rubric-based analysis in recent years:

• Interview transcripts
• Performance task data (video + transcript)
• Observations of program activities
• Student work completed during a program
• Grantee project proposals
• Program documentation
• Ethnographic-style observation notes
• Grantee reports
• Asking managers to apply the rubric to their work, with validating conversations afterward

Heck, once I applied a rubric to aggregate a set of scores generated from a bunch of other rubrics. (Perhaps I jumped the shark, but there was logic to this, I swear.)

---

Real World Example:

One of the things people worry about is that rubrics will feel judgmental to stakeholders. We all know how it felt to see a "Needs Improvement" on that elementary school report card. (Spelling. Mine was spelling.)

In our work with the Teen Science Cafe Network, we wrestled with this. As the program model spread, we needed to explore how such diverse sites could adhere to the core principles. We wanted to assess depth to understand strengths and challenges, not to judge.

We used a lot of strategies to mitigate the fear of passing judgement or getting data that reflected "grade seeking" over reality.

We used a careful rubric development process. Descriptions had to be rooted in the language program leaders use so they would see their experience in the levels. We used journey-focused labels. We tried to normalize all points on the continuum by making the top level aspirational, but not expected. And we are still creating supports and process for how it can get routinely applied.

As we tested with early sites, we asked how the process of applying the categories to their program felt. Universally, testers said reflective, but not judgmental. As one reported:

When it hits the mark, a rubric can be both an evaluative tool and a roadmap. In this case, for people looking to do this kind of program, the rubric isn't just an assessment tool, but a way to make a path to success concrete, progressive, and attainable.

​

Anyone else as smitten with rubrics as I am? Still feel like they're too judgy? Hit reply and tell me!

Want More about Rubrics?

A few years ago, we published an article delving into the process and theory of developing rubrics in education. In our case it was to assess data literacy skills. This was in undergraduate settings, but the skills apply more broadly.

Wanna really geek out about the art and science of rubric development? Bring on the Publication!

On the Road Again

We had a great time at ASTC's annual conference. It was amazing reconnecting with so many of you in person!

Where to next?

NAAEE (November 7-9, Pittsburgh, PA): I am so excited for a conference where travel entails no more than crossing a river. Coming to the Steel City? Need tips or must-dos while you're here? Email me and let's meet up!

Grant Deadline Reminder

Those IMLS deadlines are coming at us like a freight train (November 15). And NSF AISL won't be far behind (January 8)s.

If you're still struggling with sorting out that evaluation or research component, you're not alone!

Still looking for a research or evaluation partner for your project? Set up a time to chat and see how we can help. Let's talk!

P.S. Got a question you'd like us to answer in an upcoming newsletter? Hit reply and tell me what's on your mind!

P.P.S. Get this email from a colleague? Sign up to get your very own copy every month.

Why the "Evaluation Therapy" Newsletter?

The moniker is light-hearted. But the origin is real. I have often seen moments when evaluation causes low-key anxiety and dread, even among evaluation enthusiasts. Maybe it feels like a black-box process sent to judge your work. Maybe it’s worry that the thing to be evaluated is complicated, not going to plan, or politically fraught. Maybe pressures abound for a "significant" study. Maybe evaluation gets tossed in your "other duties as assigned" with no support. And so much more.

Evaluation can be energizing! But the reality of the process, methods, and results means it can also feel messy, risky, or overwhelming.

I've found that straightforward conversation about the realities of evaluation and practical solutions can do wonders. Let's demystify the jargon, dial down the pressure, reveal (and get past) barriers, and ultimately create a spirit of learning (not judging) through data. This newsletter is one resource for frank talk and learning together, one step at a time.

​Learn more about JSC and our team of evaluators. Or connect with us on LinkedIn: