beneficiary-counting-pp

With the increased attention to performance accountability, there has understandably been much focus on the measurement of impact indicators and higher level results. However, one should not underestimate the importance of and challenges for measuring basic output-level indicators, such as people reached by services (AKA “beneficiaries”). This blog will examine this, focusing on the challenges presented by direct, indirect and double counting. It will conclude that there is no universal “recipe” for accurately counting people reached, but approaches can be adapted according to organizational and operational context.

A good place to start this discussion is why counting people reached is important. As posted on one popular listserv, Pelican Initiative: Platform for Evidence-based Learning & Communication for Social Change: “How is counting numbers improving the quality of appropriate provision of services?” Indeed, “counts” are not enough – we want to know what difference is being made, and whether the return on investment is worthwhile. However, output counts help to “add-up” and triangulate with other data sources to determine the worth or utility of (evaluate) interventions. Output counts such as people reached can be important measures (when reliable) to assess evaluation criteria such as coverage and efficiency, and contribute to the evaluation of effectiveness and impact.

quote-1

Evaluation aside, the ability to count the people reached by project and programs is important for monitoring and managing their implementation, informing future planning and allocation of resources.  In fact, in my experience, counting people reached is one of the basic measures I do not need to impress on program managers because they need/want to do this to inform their service delivery.

Another consideration to clear up is the “what to call the people we count.” There is a growing critic of the term “beneficiary” in development in humanitarian relief, which I address in my companion blog, “Beneficiary Revisited.” In short, there is an important difference between measuring outreach or coverage of people reached by services, versus desired, higher levels changes in the behavior (outcomes) or condition (impact) of people reached. One should not confuse or conflate measurement by combining measures of different levels of results – people reached by services (outputs) versus those positively impacted (outcomes/impact).

Semantics aside, we should not underestimate the challenges encountered when measuring people reached, (service outreach). One such challenge is the disaggregation of service recipients. This can include their demographic characteristics; e.g. sex, gender, age, ethnicity, disability, and vulnerability. The importance of these traits and how to best disaggregate for them will depend on the implementing program focus, available resources and expertise, and other contextual factors. While demographic disaggregation is not the focus of this blog, it is nevertheless important to flag as a critical consideration when designing an M&E system for counting people reached.

The remainder of this blog will focus on the challenges of counting people reached presented by direct, indirect and double counting…

quote-1

Direct & Indirect Recipients

Another important type of disaggregation for measuring people reached is that between direct and indirect recipients of services. It is first important to define what is to be measured. However, there is no industry-recognized standard for what is meant by direct and indirect recipients, (and for that matter the names of these categories of people, as discussed above). Recognizing this…

Direct recipients can be defined as countable recipients of services by the service provider at the delivery point. In contrast, indirect recipients cannot be directly counted because they receive services apart from the provider and delivery point, (e.g. people listing to an HIV/AIDS awareness radio program). Delivery point refers to a location where a provider delivers services directly to people. This can be stationary, as with a nurse at a health clinic, or mobile, as with a roving nurse providing vaccinations at households. The key element is that the provider is present to verify delivery of service.

Based on this definition of indirect recipients, we need to acknowledge that any measure of indirect recipients is only an approximation, (because they cannot be verified in-person by the service provider). For example, the average listening audience for a radio program (i.e. an HIV/AIDS awareness-raising project) in a certain region and time of the day is an estimation based on marketing research. In short, accurate measurement is limited, and we must acknowledge the difference between “evidence” versus “proof.”

As such, an important consideration for the measurement of indirect recipients is the degree to which counts are based on assumptions that are too indirect and/or unreliable. Indeed, this is a “judgement call,” and will vary from organizational and operational context. For instance, I know of an organization that counts not only students reached by a messaging campaign through attendance at school presentations (direct recipients), but extrapolates to include indirect counts of student household members, (based on the sum of the number of students multiplied by the average household size for the region).  Depending on stakeholders involved, this assumption of messaging to family members may be perceived as unreliable. However, if students are given a homework assignment to interview one or more family member related to the messages provided at school, then this may be considered reliable enough to count family members. Is this completely accurate? No – it is an approximation. Firstly, only family members from students completing the homework assignment should be counted; secondly, there may be instances where a student “fakes” an interview for credit for the homework assignment.

Table 1 provides some other examples of different service types and how direct and indirect service recipients can be counted. The examples are illustrative, and as noted, specific protocol and rational for counting indirect recipients will vary according to context, considering the balance of reliability and legitimacy with stakeholders (e.g. donors) with what is possible given resources and technical expertise.

TABLE 1: Examples of Direct Recipients versus Indirect Recipients
Service Type
Direct Recipients
Indirect Recipients
Counting Rationale
Water & Sanitation
Households that receive wells and toilet facilities
Not applicable
Count as direct recipients the number of HH members in HH receiving either or both wells and toilet facilities.
Community HIV awareness radio broadcasts
Not applicable
1. Community members listening to the broadcasts.
2. Community members receiving secondhand messaging from firsthand listeners.
Count radio listening audience as indirect recipients using informed estimates as indirect recipients; Do not count secondhand, indirect recipients of messages.
Taxi driver road safety program
Participating taxi drivers
Passengers that receive taxi driver reminders to use their seatbelt.
Count taxi drivers as direct recipients, but not passengers, unless a reliable means to verify passengers as indirect recipients of messaging (i.e. automated recorded message when driver starts taxi meter)
Cash for work (CFW) program cleaning up a community after an earthquake
CFW participants receiving cash for community cleanup work
Community members benefiting from environmental sanitation provided through CFW program.
Count both household members and community members.
Construction of a community hospital
Community members treated at the hospital
Community members benefiting from presence of local health services, (e.g. increased health security and reduced burden for provision of services to family members).
Count treated community members as direct recipients and catchment of community population as indirect recipients.
Establishment of a community disaster Early Warning System
Community population if EWS is used and works.
Community members benefiting from presence of EWR for potential future disasters.
Count community population as direct recipients if EWS is used and works and catchment of community population as indirect recipients.

Double Counting

Another important challenge for counting people reached, whether direct or indirect, is that of double counting, or counting the same person reached by a service provider (organization) more than once in the same reporting period. Double counting is to be avoided because it inflates the count of people reached and is therefore misrepresentative.

Table 2 provides a simple (and extreme)  example of how double counting can inflate the total number of people reached beyond so that it can actually be more than total population. Double counting occurs in this example because the organization aggregates the counts of people reached by each program in its recovery operation, rather than controlling for individuals reached by multiple programs (service types) and adjusting the total count to avoid counting them more than once during the reporting period.

TABLE 2: Double Counting Example
Xland Disaster Recovery Operation, (reporting period 1/12/2015 – 12/31/2015)
Xland population
500,000
People Reached by
Service Type
Food relief items
400,000
Non-food relief items
300,000
Shelter provision
200,000
Water/Sanitation provision
200,000
Vector-borne disease prevention
200,000
Psychosocial services
100,000
Total number people reached by Xland disaster recovery program. (with double counting mistake)
1400,000

Table 2 is an illustration of double counting that can occur with the delivery of multiple services from one provider. Table 3 summarizes this and other common causes of double counting, and it is worth noting that sometimes an organization confronts a combination of these challenges.

TABLE 3: Common Double Counting Mistakes
Mistake Type
Example
1.       Double counting individuals who receive multiple services from the same provider
Example: Organization reaches 100,000 individuals in a disaster recovery operation, and within this group 50,000 also participated in health care program provided by the organization; the total people reached is 1,000, not 1,400.
2.       Double counting individuals who receive more than one service over time.
Example: Individuals receiving HIV testing and counseling at a health center in April, July, and November are counted for each visit.
3.       Double counting individuals who receive services at multiple delivery points.
Example: Individuals attending a family planning presentation on the west side of town, and later on the east side provided by the same organization is counted more than once.
4.       Double counting individuals who are both direct and indirect recipients.
Example: Individuals attending a first aid class and also receiving indirect first aid messages in their community (e.g. billboards, radio or TV) from the same organization are counted more than once.

Double counting can be avoided by establishing monitoring systems that carefully track people reached by service type, provider, delivery point, and time period. Oftentimes, such monitoring is already a regular part of program information management systems to understand the target population’s needs, allocate people and resources, and coordinate services and partners. Some helpful points to keep in mind include:

  1. Anticipate and plan for instances where double counting is more likely. For example, if there is a logical framework, review program components and indicators at each level to help identify when certain target populations, services, or providers may overlap.
  2. When possible, use a tracking system that can uniquely identify each individual receiving a service so that at the end of the reporting period there are accurate lists of individuals (by name and/or ID number) that can be used to make counts and adjust total counts across time, place, provider and service type.
  3. When working with households, determine from the outset whether individuals will be counted, or calculated by multiplying the number of households reached by average household size. If counting individuals AND households, make sure that the interventions do not overlap the different counting methods.
  4. Mapping the program landscape can help reduce double counting and support the use of catchment counts when appropriate. This involves the use of maps (paper or computerized), to represent the locations of services, recipients and providers. When the likelihood is high of a given target population receiving at least one service over time within the service delivery area, the total population can be counted as people reached.

In summary…

There is no “magic recipe” to count people reached. The ability to reliably control for double-counting will depend on a variety of factors according to the organizational and operational context; i.e. an organization’s scale and scope of services, and the availability of resources and technical expertise available for aggregating counts of people reached. For example, it may be possible with an organization delivering relief items to use bar codes to accurately identify and track aid recipients to avoid double counting…when access to such technology is affordable and practical.

Another important consideration is the return on investment for controlling for  discussed challenges to accurately report on people reached. This will depend on the key stakeholders, including not only those requiring such data (e.g. donors), but also the program/project tasked with collecting the data, and the degree to which such efforts can affect the delivery of services to the very stakeholders that organizations seek to serve (e.g. the target population). We do not want the “accountability tail to wag the dog,” meaning we do not want measurement for M&E to burden the very service delivery (programming) that it is supposed to support.

beneficiary-counting-pp

In light of the above “reality check,” counting the people we serve may not always be a priority or even practical. For example, one recent publication, Forced Displacement, Go Figure! Shaking the Box of Profiling IDP Situations (Chemaly et. al. 2016), states, “The focus of Profiling IDP [internally displaced people] situations should be not on accurate numbers but rather on displacement trends or ranges. Relevance and reliability are more useful than precision.”

Acknowledgements – While this blog represents my own views, they are informed by my work as a Senior M&E Advisor with the International Federation of Red Cross and Red Crescent Societies (IFRC), for which I developed the measurement guidelines for its proxy indicators for the Federation-wide Databank and Reporting System, including the indicator for people reached. The cartoon is illustrated by Julie Smith and provided courtesy of the IFRC, and the picture at the start of this blog was taken by myself in Laos in 2010.