Bringing Context Into Focus

I evaluate two university programs funded through the same federal initiative. In many ways, they look remarkably similar.

Both are designed to contribute to national security by preparing students for cybersecurity and AI careers in public service. They draw on many of the same kinds of resources and activities—scholarship support, cybersecurity education, professional development, experiential learning, and connections to public service careers—and they are working toward many of the same outcomes.

On paper, it would be reasonable to assume that evaluating them should look pretty similar, too.

But programs don’t operate on paper. They operate in context.

The two universities are situated in very different educational, regional, and professional environments. At one, students live in the Baltimore-Washington region, where the federal government and federal employment are a visible part of the professional landscape. At the other, a large proportion of students are the first in their families to attend college, the surrounding region faces different employment conditions, and Washington—and the federal employment landscape students are preparing to enter—is nearly 3,000 miles away.

Those differences made me think more carefully about what it means, in each of these programs, to prepare students for public service careers. For some students, federal employment may already be a familiar professional world. For others, it may represent an entirely new professional environment—and a career path that will likely require moving far from familiar surroundings, family, and established support networks.

The intended outcome may be the same. But what students need—and what they experience along the way—may not be.

Theory Tells Us Context Matters

The importance of context is well established in evaluation. Across different theories and approaches, evaluators are encouraged to consider the conditions surrounding a program: its needs, resources, opportunities and constraints; the institutional and environmental factors that influence implementation; and the experiences of the people involved.

The CDC’s 2024 Program Evaluation Framework makes that expectation explicit: Step 1 is Assess Context. Understanding the setting in which a program operates helps us determine what is important to know, what success looks like, and how findings should be interpreted. In other words, context isn’t simply background information. It can shape the evaluation itself.

Despite its emphasis in the literature, assessing context in practice can feel a bit like navigating the night sky—at least for this evaluator. There is a lot to see. The challenge is knowing where to look.

Knowing that context matters is one thing. Figuring out what matters in this context is the work.

Grounding the Evaluation in Context

There isn’t a single method for doing that work.

In evaluating these two programs, I’ve drawn on program documents, existing research and data, key informant interviews, and site visits. Each helps me see something different about the program and the environment in which it operates.

But context isn’t simply a collection of characteristics we can document. It is living, changing, and experienced by the people within it. Understanding it requires listening to people who know the setting, paying attention to what they identify as important, and sometimes looking beyond our usual sources of evidence.

Take the two programs I’m evaluating. Both include activities such as mentoring, networking, career preparation, and opportunities to interact with government employers. On a logic model, those activities could look almost identical.

But what are they expected to accomplish in each context?

For a student already familiar with federal employment, interacting with a government employer might primarily provide a new professional connection. For a student with little exposure to that environment, the same opportunity might also make federal careers more understandable, provide insight into navigating that professional world, or make a career possibility more concrete.

The activities may be the same. The intended outcomes may be the same. But what happens between the two can be shaped by context.

Understanding context also requires recognizing that we make choices about where to look. The people we speak with shape what comes into focus. So do our own experience, disciplinary training, and assumptions.

Working across the two programs makes this especially visible to me. What I learn in one setting helps me ask better questions in the other—but it can also lead me to assume that something important in one context will be equally important, or mean the same thing, in another.

This is where thinking about Ways of Knowing in Evaluation can be useful. It shifts the question from simply What do I know about this context? to How have I come to know it? Whose perspectives have shaped my understanding? What has my disciplinary knowledge led me to notice? What assumptions am I bringing from my own experience? And what might I be missing?

More information doesn’t automatically mean better understanding.

Context Can Change How We Measure Success

I learned this years ago while helping develop and evaluate a First Nations (Indigenous) nursing program. The program made nursing education possible for students whose family and other personal responsibilities prevented many of them from attending nursing programs outside their community. Graduates also helped increase the community’s access to quality healthcare.

One seemingly straightforward indicator of success was graduation within the expected two-year timeframe. From a disciplinary standpoint, it made sense: students entered a two-year program, so completing it within two years seemed like an appropriate marker of success.

But the data indicated that students were taking longer to graduate than students in comparable programs. Viewed through that measure alone, the program could have appeared unsuccessful.

Other ways of knowing complicated that interpretation.

Conversations with community members, site visits, and attending a graduation ceremony helped me better understand the context surrounding students’ experiences, including the family and community responsibilities many were balancing. Those responsibilities were not peripheral to students’ lives; they were part of the context in which education was taking place.

And they didn’t disappear simply because nursing education was now available closer to home. Some students needed to step away from their studies and return later.

The important question wasn’t whether students finished within two years. What mattered to success was that they successfully completed the program—and that shifted my attention to the supports that were most important in helping them do so.

So I reconsidered the measure. Rather than relying on two-year graduation as the primary marker of success, I looked at completion over a longer period. That shift also brought the role of program supports into clearer focus: helping students stay connected, reaching out when they stepped away, and supporting their transition back into the program when they were ready to return.

Without that contextual understanding, a valuable program could have looked like a failure. Looking at completion over a longer period gave us a more appropriate way to assess success—and helped us better understand what the program needed to do to support it.

What Changes?

Understanding context can shape decisions throughout an evaluation—how we understand the program, the questions we ask, the evidence we seek, the measures we choose, and the judgments we make.

And our understanding of context shouldn’t be treated as settled. We can draw on multiple sources and perspectives and still learn something that causes us to reconsider what we thought mattered or what we thought we understood.

Theory can remind us to pay attention to context. Methods can help us investigate it. Different ways of knowing can help us question our understanding of it.

Sometimes context changes how we understand a finding. Sometimes it changes what we look for, how we measure success, and what we understand is needed to achieve it.

Explore Further

• CDC Program Evaluation Framework, 2024 — the framework places assessing context at the beginning of the evaluation process.
• CDC: Step 1 — Assess the Context — practical guidance for considering the setting in which an evaluation takes place.
• Ways of Knowing in Evaluation — my companion primer on what we know, how we know, and why it matters in evaluation practice.