With the increased attention to performance accountability, output indicators, like the number of people reached by services, have understandably taken a backseat of impact indicators used to determine what difference has been made. However, one should not underestimate the importance of and challenges for measuring basic output-level indicators, such as people reached by services (AKA “beneficiaries”). This blog will examine this, focusing on the challenges presented by direct and indirect recipients, as well as double counting. It will conclude that there is no universal “recipe” for accurately counting people reached. Instead, approaches can be adapted according to organizational and operational context, with attention to the legitimacy and degree of utility.
Firstly, one may ask why counting people reached is important. As posted on one the Pelican Initiative listserv, (exchanging ideas related to evidence-based social change): “How is counting numbers improving the quality of appropriate provision of services?” Indeed, “counts” are not enough – we want to know what difference is being made, and whether the return on investment is worthwhile.
However, output counts help to “add-up” and triangulate with other data sources to determine the worth or utility of interventions (evaluation). Output counts such as people reached can be important measures (when reliable) to assess evaluation criteria such as coverage and efficiency, and contribute to the evaluation of effectiveness and impact.
|We need to acknowledge that the measurement of higher level results (impact) is as reliable as the measurement of basic, lower level results (outputs and counts of outreach) that help determine higher level change.|
Evaluation aside, the ability to count the people reached by project and programs is important for monitoring and managing their implementation, informing future planning and allocation of resources. In fact, in my experience, counting people reached is one of the basic measures I do not need to impress on program managers because they need/want to do this to inform their work.
Another consideration to clear up is the, “what to call the people we count.” There is a growing critic of the term “beneficiary” in development in humanitarian relief, which I address in my companion blog, “Beneficiary Revisited.” In short, there is an important difference between measuring outreach or coverage of people reached by services, versus desired, higher levels changes in the behavior (outcomes) or condition (impact) of people reached. One should not confuse or conflate measurement by combining measures of different levels of results – people reached by services (outputs) versus those positively impacted (outcomes/impact).
Semantics aside, we should not underestimate the challenges encountered when measuring people reached, (service outreach). One such challenge is the disaggregation of service recipients. Demographic disaggregation includes characteristics such as sex, age, disability, ethnicity, education and income. Demographic disaggregation is not the focus of this blog, but it is nevertheless important to flag as a critical consideration when designing an M&E system for counting people reached.
The remainder of this blog will focus on the challenges of counting people reached presented by direct, indirect and double counting…
|The challenges of direct, indirect and double counting can be particularly formidable for organizations operating in complex setting, involving multiple programs overlapping in time, location, and population groups.|
Direct & Indirect Recipients
One fundamental way that counts of people reached can be disaggregated is by whether they received the service directly or indirectly. There is no industry-recognized standard for what is meant by direct and indirect recipients, but I have my professional opinion…
Direct recipients can be defined as countable recipients of services by the service provider at the delivery point. In contrast, indirect recipients cannot be directly counted because they receive services apart from the provider and delivery point, (e.g. people listening to an HIV/AIDS awareness radio program). “Delivery point” refers to a location where a provider delivers services directly to people. This can be stationary, as with a nurse at a health clinic, or mobile, as with a roaming nurse providing vaccinations at households. The key element is that the provider is present to verify delivery of service.
Based on our definition of indirect recipients, we need to acknowledge that any measure of indirect recipients is only an approximation, (because they cannot be verified in-person by the service provider). For example, the average listening audience for an awareness-raising radio program in a certain region and time of the day is an estimation based on marketing research. In short, accurate measurement is limited, and we must acknowledge the difference between “evidence” versus “proof.”
An important consideration for the measurement of indirect recipients is the degree to which counts are based on assumptions that are too indirect and/or unreliable. Indeed, this is a “judgement call,” and will vary from organizational and operational context. For instance, an organization counts not only students reached by a messaging campaign through attendance at school presentations (direct recipients), but extrapolates to include indirect counts of household members of students, (based on the sum of the number of students multiplied by the average household size for the region). Depending on stakeholders and context, the assumption that students will provide messages to their family members may be questionable.
However, if students are given a homework assignment to interview one or more family member related to the messages provided at school, then this may be considered reliable enough to count family members. Is this completely accurate? No: it is an approximation. Firstly, only family members from students completing the homework assignment should be counted; secondly, there may be instances where a student “fakes” an interview for the homework assignment.
Table 1 below provides some other examples of different service types and the counting rationale for direct and indirect service recipients can be counted. The examples are illustrative, and the justification will vary according to context, balancing reliability with legitimacy, and what is possible given resources and technical expertise.
|TABLE 1: Examples of Direct Recipients versus Indirect Recipients|
|Service Type||Direct Recipient||Indirect Recipient|
|Provision of water and/or sanitation facilities||Count the household population served by the provision of wells and latrines.||Indirect recipients are not applicable. [One may be tempted to include the total community populations, justifying that improved personal hygiene – e.g. reduced open defecation – reduces communicable diseases and therefore serves everyone. However, this may be considered too much inference.]|
|Taxi driver road safety program||Do not estimate counts of taxi passengers assumed to receive taxi driver messages, such as reminders to use their seatbelt, unless there is a reliable means to verify passengers and messaging, such as an automated recorded message when the driver starts the taxi meter.|
|Hospital (or health clinic) services||Community members treated in the hospital should be counted as direct recipients. However, to avoid double counting, only count each person once per the reporting period, even if they have received more than one medical service (e.g. a sequence of vaccinations). This will require reliable data management systems for patient registration by individual identities (whether names or codes) at each facility, and the linking of this data between facilities for aggregation.
If the hospital program has objectives that target any employees as service recipients (e.g. through gainful employment, training, etc.), they would also be counted as direct recipients, as well as their household members.
|The fact that the hospital serves a certain population is relevant, but it is not recommended to estimate specific ounts of community members assumed to indirectly benefit from increased health security by the presence of the hospital.|
|Community radio and TV broadcasts||Direct recipients are not applicable because people have not yet received services for RCRC personnel to verify.||Estimate counts of indirect recipients based on estimates of the radio listenership and television viewership. Oftentimes, these audiences can be estimated from marketing information available for commercial purposes from the broadcasting service. Do not estimate counts of indirect recipients receiving second-hand messaging from first-hand listeners, (people who did not listen or view the radio or TV messaging themselves, but learned about it from others who did).|
|Establishment of an early warning system||Count as direct recipients the population of those communities (if any) for which the EWS is triggered during the reporting period by a disaster and works as intended.||If the EWS is not triggered for an actual disaster, then community populations can be counted as indirect recipients in those communities for which the EWS is successfully tested during a disaster simulation.|
|School educational program||Count high school students based on class attendance as direct recipients, but only once, regardless of the number of presentations they may have attended during the reporting period.||Do not count family members who may learn second-hand from students about the presentation as indirect recipients. [However, if there is a school homework assignment related to the RCRC principles in which students interact with family members, then these household members can be counted as indirect recipients.]|
|Livelihoods household program||Count people and their household members as direct recipients each year. In this instance, it is acceptable to include household members as direct recipients though they may not all be present at the service delivery point because it is reasonable and reliable to conclude that household members are reached by the intervention.||Indirect recipients are not applicable.|
|Video messaging through YouTube||The accurate measurement of indirect recipients for initiatives using social media can be challenging. This particular example is one for which there is NOT an easy way to determine people reached (viewers) of the video. YouTube Analytics can be used to determine how many views through a Watch-time Report, but this can include repeat views from the same person. YouTube analytics can also be used to generate a Devices Report of the different devices and operating systems that viewers use to watch your videos. However, this still does not provide distinct counts of individual people reached. Nevertheless, they, and other YouTube metrics can help better understand service outreach. Learn more at YouTube Watch-time Reports.|
Another important challenge for counting people reached, whether direct or indirect, is that of double counting, or counting the same person reached by a service provider (organization) more than once in the same reporting period. Double counting is to be avoided because it inflates the count of people reached and is therefore misrepresentative.
Table 2 provides a simple example of how double counting can inflate the total number of people reached beyond the total population. Double counting occurs because the organization aggregates the counts of people reached by each program in its recovery operation, rather than controlling for individuals reached by multiple programs (service types) and adjusting the total count to avoid counting them more than once during the reporting period.
|TABLE 2: Double Counting Example|
|Xland Disaster Recovery Operation, (reporting period 1/12/2015 – 12/31/2015)|
|People Reached by
|Food relief items||400,000|
|Non-food relief items||300,000|
|Vector-borne disease prevention||200,000|
|Total number people reached by Xland disaster recovery program.||1400,000|
Table 2 is an illustration of double counting that can occur with the delivery of multiple services from one provider. Table 3 summarizes this and other common causes of double counting. Note that sometimes an organization confronts a combination of these challenges.
|TABLE 3: Common Double Counting Mistakes|
|1. Double counting individuals who receive multiple services from the same provider||Example: Organization reaches 100,000 individuals in a disaster recovery operation, and within this group 50,000 also participated in health care program provided by the organization; the total people reached is 1,000, not 1,400.|
|2. Double counting individuals who receive more than one service over time.||Example: Individuals receiving HIV testing and counseling at a health center in April, July, and November are counted for each visit.|
|3. Double counting individuals who receive services at multiple delivery points.||Example: Individuals attending a family planning presentation on the west side of town, and later on the east side provided by the same organization is counted more than once.|
|4. Double counting individuals who are both direct and indirect recipients.||Example: Individuals attending a first aid class and also receiving indirect first aid messages in their community (e.g. billboards, radio or TV) from the same organization are counted more than once.|
Double counting can be reduced by establishing data management systems that carefully track people reached by service type, provider, delivery point, and time period. Oftentimes, such systems are already a regular part of program information management to understand the local context (needs), allocate people and resources, and coordinate services and partners. Some helpful points to keep in mind include:
- Anticipate and plan for instances where double counting is more likely. For example, if there is a logical framework, review programme components and indicators at each level to help identify when certain target populations, services, or providers may overlap. Related, compare logical frameworks between projects/programmes to identify target population overlap.
- When possible, use a tracking system that can uniquely identify each individual reached by a service, so that at the end of the reporting period there are accurate lists of individuals – (by name and/or ID number, recoding their sex and age as well as other factors that can be used to analyse and inform programmes such as disability status or at-risk groups) – that can be used to make and adjust counts across time, place, provider and service type.
- When working with households, determine from the outset whether individuals will be counted, or calculated by multiplying the number of households reached by average household size. If counting individuals AND households, make sure that interventions do not overlap the different counting strategies.
- Mapping the programme landscape can help reduce double counting and support the use of catchment counts when appropriate. This involves the use of maps (paper or computerized), to represent the locations of services and providers. When it is reliable that all individuals in a given target population will receive at least one service over the given time within the service delivery area, the total population can be counted as people reached.
In summary, there is no “magic recipe” to count people reached. The ability to reliably control for double-counting will depend on a variety of factors according to the organizational and operational context; i.e. an organization’s scale and scope of services, and the availability of resources and technical expertise available for aggregating counts of people reached.
For example, it may be possible for an organization to minimize double counting when reaching people with multiple services over time and place by using bar codes to uniquely identify and track individual service recipients; however, this may be possible when access to such technology is affordable and practical.
Conclusion – be realistic and responsible
An important consideration for reliable counts of people reached, (as well as any data to be collected and reported), is the given capacity to manage data along the “data lifecycle.” Data management concerns a range of processes, from data collection, verification, organization, storage, analysis, and presentation, to the eventual “retirement” of data. One should not underestimate the capacity requirements for reliable and accountable data management – not just counting people reached, but also other indicators.
For instance, data collection guidance, templates and forms may be required at the field level, as is training data collectors (whether using paper or handheld digital device). From the field level, data management systems may then be required to aggregate, safely store, access and “roll-up” data from the local and regional level up to country and global offices. Other considerations include quality assurance checks that cross-check and clean data.
With the above in mind, careful attention should be given to what is feasible to reliably collect and report given the existing capacities for a specific program or project. This includes not only to existing systems for data management, but also the human and financial resources required operate them. We do not want the “accountability tail to wag the dog,” meaning we do not want M&E measurement to burden the very service delivery (programming) it is supposed to support.
 This can include mobile data collection applications (software), such as Open Data Kit (ODK), Kobo, and Magpi.
A final message – be sure data is managed in a responsible manner that safeguards the dignity, respect and privacy of the individuals, organizations, and other key groups from which we collect data. For people reached, this includes balancing key principles related to people’s right to be counted, informed consent, data sharing and transparency, and ensuring that we act in the best interests of the people we count to serve.
Postscript – While this blog represents my own views, they are informed by my prior work with the International Federation of Red Cross and Red Crescent Societies (IFRC), for which I led the development of the measurement guidelines for its proxy indicators for the Federation-wide Databank and Reporting System, including the indicator for people reached. The cartoon is illustrated by Julie Smith and provided courtesy of the IFRC. The picture at the start of this blog was taken in Laos with consent from the family to be used online.