profile

The Evaluation Therapy Newsletter

STEM career outcomes? It's the journey, not the destination.


Hi Reader,

It's been quite a week, hasn't it? I've found myself thinking a lot about education in recent days.

Particularly, I'm thinking about education that develops and hones our critical thinking skills. Skills to hold and consider multiple ideas at the same time. Weigh evidence and multiple variables, particularly when they contradict. Seek multiple sources to validate information. Question potential bias in evidence sources. An education that should start early (elementary-aged kids can do this) and continue into adulthood.

Educator Me calls it critical thinking. Folksy Me calls it a well-oiled Bullshit Detector. Call it what you like, but I feel like we need a lot more of that skill in society.

In the spirit of raising future adults who sniff out the BS and make positive change, this month we'll grapple with ways that out-of-school learning can meaningfully contribute to career trajectories. (Spoiler alert: don't ask about careers.)

Also included is:

  • Related resource: Science Capital explained
  • Unrelated resource: hashing out the role of evaluation within NSF learning research

Hang in there,

Jessica

9 to 5, what a way to make a living.

If you work around STEM education long enough, you'll encounter a funder who wants to promote "STEM career" outcomes. With high schoolers. With middle schoolers. Heck, I'm sure someone out there is trying to move a first-grader's career needle from fire fighter to astronomer.

Yes, we absolutely need more (and more diverse) people in STEM-based professions. But framing program outcomes around some far-future career is way too big and vague to be plausible from one program.

We're all adults here. With jobs. When you were 15, would you have described anything close to the job you have now? For 97%* of us, our job was not on our radar at 15. Heck, for many folks in STEM, their job didn't exist when they were 15.

*Note: I 100% invented that statistic.

With the exception of my childhood bestie, who knew she would be a doctor when we were giving our Barbies dream careers, most everyone I know muddled through education and adulthood until we found our thing. And our next thing. And our thing after that.

Career paths are complicated, fluid, and involve a lot of external factors. When you look at yourself, you know this. But, sometimes, when we look at tweens and teens, we forget.

Angie and I have been reading and coding 300+ teen reflections about what they envision for their future. It was enlightening. Answers were less "I want to be a doctor" and more Lloyd Dobler:

What's the problem?

It's a problem if it wastes valuable program time. If educators feel pressured to cram a non-sequitur Career Day into programs, it's at the expense of the learning experiences they are good at doing.

It's a problem if it pushes you to measure the wrong outcomes. Force the issue, and you're wasting valuable evaluation energy measuring a thing that is, at best, predictively unsound. And at worst, you miss documenting the real benefit you are creating.


What do we do?

First, as a field, we need to get a grip. No program experience, no matter how amazing, is going to single-handedly change the career path of most students. Let's not pretend we expect it to.

Will there be that one amazing story of the student for whom it literally changed everything? Sure!

Will there be others for whom this program was one of several life experiences that contributed to taking a particular path? Absolutely!!

But if your questions are about most of the kids, the road will be messy, winding, and personal - and you can't see the endpoint from here.

What can I do instead?

Think about career trajectories like a Candy Crush game. You can't escort them through every level of the saga. But you can equip them with a specific "booster" to help them progress through the next stages. You can identify the right boosters for their level -- because needs are different at Level 25 and Level 4,276. Ultimately, it will be up to them, amidst a myriad of life experiences, to put it all to use.

Get really specific about what types of supportive boosters your program fosters. Then you can make the case of how those knowledge, skills, and mindsets are key to enabling choices, confidence, and success at every turn in the long journey ahead.

Some ideas to get you started:

  • Meeting role models: pros who look like me, sound like me, also didn't take a straight path to science
  • Incredibly psyched about a topic: momentary interest is OK, but fleeting; can you get to psych-itude that holds up over time?
  • Aware of more / more specific careers in a field: and/or desirable attributes of certain careers (close to home, good pay, doesn't require endless schooling, uses tools/skills I like)
  • Understanding what a career actually does: careful, this can backfire when youth realize that work is, you know, work.
  • Opportunities to level up: offering and them taking a meaningful next step
  • Transferrable Skills: technical, interpersonal, self-direction, resilience
  • Identity: self-concept, how you see yourself

And here is my obligatory mention that there's loads of research literature on this topic, if you want to get nerdy and deep-dive.


Real World Example:

Let's talk about what Angie and I have been finding in data from ~150 teens who took part in Build a Better Book engineering internships using engineering design and Maker Space tech to develop games, books, toys, and tools to be accessible for blind and low-vision users.

When we asked HS interns to tell us, flat-out, what they were thinking about for future study or careers, most gave nearly identical answers before and after their internship.

The few answers that changed? Half went from totally clueless to considering engineering! But the other half went from being certain about STEM to feeling totally clueless.

Those questions asked about the wrong thing.

When we centered on the skills and mindsets developed, we saw the impact:

  1. Shifting perceptions of what engineering work is
  2. Higher confidence in their engineering design skills
  3. Growth in workplace skills and mindsets -- communication, creativity, self-direction, critical thinking, collaboration, resilience, professionalism
  4. Understanding how to use universal design in engineering
  5. Facility with technical skills and tools
  6. Uptick in sense of self as an engineer (identity)

In our project's game of Candy Crush, these were six super color bombs that teens walked away with. In whatever comes next, we're hopeful their skill gains, identity shifts, and know-how will keep being used.

And, let's be honest, we might follow-up with them after a year or two to ask.

So, what skills, mindsets, knowledge, or simple aha moments do you create really well? Reply with your thinking!

Science Capital

If you're grappling with goals like these, may I introduce you to the concept of "science capital"? Developed by researchers in the UK (museum folks!), it's a useful framework.

It's not a simple framework. They've worked hard to unpack many different factors -- I'd call them "power boosters" -- in life that contribute toward an adult who engages and identifies with science.

Since they've unpacked a lot of those factors, it can be helpful to spark thinking about your program's specific role in that bigger landscape.

Want their user-friendly rundown of Science Capital?

What is evaluation in a research project?

It's that time of year again! If you are involved in NSF research projects, you've likely wrestled with this question. When the research plan is examining learning outcomes of the program, and the solicitation also asks for external evaluation... what, exactly, should the evaluation be doing?!?

A while back, I discussed this topic in an interview with CADRE, alongside Dan Hanley of Western Washington University. We're both switch-hitters in the world of educational research and evaluation, so we see the conundrum from both sides.

It's a fun conversation (there is a sad trombone sound) about an admittedly wonky topic.

WTF is the role of evaluation in a research project?

P.S. Got a question you'd like us to answer in an upcoming newsletter? Hit reply and tell me what's on your mind!

P.P.S. Get this email from a colleague? Sign up to get your very own copy every month.

Why the "Evaluation Therapy" Newsletter?

The moniker is light-hearted. But the origin is real. I have often seen moments when evaluation causes low-key anxiety and dread, even among evaluation enthusiasts. Maybe it feels like a black-box process sent to judge your work. Maybe it’s worry that the thing to be evaluated is complicated, not going to plan, or politically fraught. Maybe pressures abound for a "significant" study. Maybe evaluation gets tossed in your "other duties as assigned" with no support. And so much more.

Evaluation can be energizing! But the reality of the process, methods, and results means it can also feel messy, risky, or overwhelming.

I've found that straightforward conversation about the realities of evaluation and practical solutions can do wonders. Let's demystify the jargon, dial down the pressure, reveal (and get past) barriers, and ultimately create a spirit of learning (not judging) through data. This newsletter is one resource for frank talk and learning together, one step at a time.

Learn more about JSC and our team of evaluators. Or connect with us on LinkedIn:

Copyright © 2024 J. Sickler Consulting, All Rights Reserved

You are receiving this email because you signed up for our newsletters somewhere along the line. Changed your mind? No hard feelings. Unsubscribe anytime.

Wanna send us some snail mail? J. Sickler Consulting, 100 S. Commons, Suite 102, Pittsburgh, PA 15212

The Evaluation Therapy Newsletter

Our monthly Evaluation Therapy Newsletter shares strategies, ideas, and lessons learned from our decades of evaluating learning in non-school spaces - museums, zoos, gardens, and after-school programs. Jessica is a learning researcher who is an educator at heart. She loves helping education teams really understand and build insights from data that they can use immediately – even those who are a bit wary of evaluation.

Share this page