Some primary evaluation users are more equal than others: Lessons from UFE experiences

DECI-4 Blog

Some primary evaluation users are more equal than others: Lessons from UFE experiences

By Ricardo Ramirez, DECI-4.

In utilization focused evaluation (UFE) and its developmental variation (UFDE), engaging primary interested users (PIUs) appears straightforward on the surface.  An important step for the evaluators is to convey their role and obligations as clearly and as early as possible.   This step means explaining that for PIUs to take on this role effectively a number of factors need to be in place: a commitment to the process and time requirements, a sincere interest in learning, a willingness to own the process and outcomes. In essence, PIUs are the owners of the evaluation design and this role takes some effort. However, many potential interested PIUs are unable to fulfill the role – often due to lack of time; or differences emerge in levels of authority and extent of commitment to the evaluation process.  The loss of commitment can be the result of many different factors, one of them is power differentials.

The power differentials affect the extent to which some potential PIUs are able to exercise the role.  The differentiation between levels of power and commitment can illuminate some (perhaps hidden) internal dynamics such as: who is really in charge, and who is not. It is only when the work begins that the differences become palpable. These dynamics can become particularly delicate in those cases in which funders become members of a PIU group.

We have learned that it is while establishing evaluation readiness that one begins to understand the actual relationships inside organizations / projects.  Often, it is not that  PIUs hide these patterns, but rather that they take them for granted because that is the reality within which they live. Other times, the differences only come to the surface under the pressure of an impending evaluation. The evaluation process is often useful because it helps make explicit that which is often implicit and, in our experience, it can surface differing perceptions within the organization.

The following are examples where differences emerged among probable PIUs -or among those who have been selected.

  • Senior management was made aware of the responsibilities of a PIU and they delegated that role to a junior manager. This delegation to a peer avoided creating differences in power or authority if a more junior staffer had been selected.
  • In several instances some PIUs were unable or unwilling to maintain a significantly active role, leaving one or two individuals in charge and thus became what we referred to as ‘light users’.
  • We initially succeeded in engaging most senior managers as PIUs, only to experience the dominating influence of a senior board member and co-founder who overruled the initial evaluation design.
  • A project partner faced some confusion between the ‘users of the project services (the audience)’ and ‘the primary users of the evaluation’.
  • In three instances, we faced the loss of users through staff turn over, in two cases the PIUs who retired were from government organizations.
  • Lastly, in one instance the uneven power balance between funder (or funding intermediary) and grantee was very evident, leaving the grantee as a lighter user.

We have concluded that PIUs are never fully independent: self-censorship to accommodate expected uses by funders, or others in control is to be expected.  This reality does not diminish the value of UFE or UFDE, but it does remind the evaluator that a design phase that engages closely with PIUs may replicate existing power differences.  While evaluators have some space to mediate these imbalances, in many instances our role is limited. Having primary evaluation users with more power than others is inevitable; the challenge is to facilitate the differences as much as possible to maximize the integration of relevant uses to meet the needs of as many USERS as possible.