Introduction
Since the late 1990s, as a culture of performance management has become ingrained in Irish public services, the pressure on public programmes to evidence their impact within standard measurement frameworks has intensified. While accountability for public funding matters, it raises a critical and underexplored question: are standard measurement tools always fit for purpose across all public programmes? What happens when they are not? And what can we learn from this?
This paper uses a recent national review of the Primary Health Care for Travellers Projects (PHCTPs) as a case study to examine these questions in the context of evaluating community development in Ireland. Specifically, it explores what happens when projects for community development work are held accountable through narrow monitoring systems that capture activity counts, while meaningful evaluation of their actual outcomes remains underdeveloped.[i] The full review report, which sets out detailed findings and recommendations, is available to download alongside this paper and provides the evidence base from which the analysis and argumentation here is drawn.[ii]
Background
The PHCTPs are a cornerstone of Ireland’s community development health model for Travellers.[iii] The PHCTPs were set up in the mid-1990s as a means of providing a collective response to what key stakeholders in the Department of Health and the then Health Boards, recognised was an unacceptable gap between Traveller health and the majority population. For over thirty years, they have been operating at the intersection of primary health care and community development, adapting and responding to the needs of Travellers. Central to the model is the peer-led role of Traveller Community Health Workers (TCHWs), who work from a social determinants of health approach within their own community to support effective engagement with health information and mainstream services, to promote and advocate for improved Traveller health, and to act as trusted intermediaries between Travellers and the health system.
An independent evaluation was requested by the Department of Health under Action 27 of the National Traveller and Roma Inclusion Strategy II. [iv] The review involved qualitative, participatory research with approximately 150 participants and was endorsed by the National Traveller Health Implementation Group (NTHIG), the Minister of State for Public Health, Wellbeing and the National Drugs Strategy, and the HSE Chief Executive Officer.
Evaluation findings
The review found that the PHCTPs are widely perceived as effective, trusted, and valuable within the community and among wider stakeholders. Most consistently, stakeholders emphasise the community development and peer-led model, the depth of long-term relationships between TCHWs and Traveller families, and the trust these relationships generate far beyond health. The centrality of relationships with families and communities is not a peripheral feature of the PHCTP model. It is core to its impact: it is the “how” through which the model is delivered and the “why” explaining its success and value in the community to date. This is consistent with an evaluation of a PHCTP in County Offaly, which found that positive health outcomes were achieved not through standard service provision but through the emotionally supportive relationships forged by community health workers, with participants describing the difference that shared cultural understanding and trust made to their willingness to engage with health services.[v] While the PHCTP model is rooted in primary health care, it produces a ripple effect, or multiplier, that extends to work vital for social change, such as challenging racism and discrimination, consciousness raising, and self-advocacy.
However, the outcomes of PHCTPs that are deemed most important by communities – collective action, advocacy, trust building, empowerment and crisis response – are almost entirely invisible within current monitoring structures of PHCTPs. Instead, a limited set of Key national Performance Indicators (KPIs) developed by the Health Service Executive (HSE) requires projects to submit a narrow range of indicators, mostly relating to specific health information activities. In response, PHCTPs have invested considerable effort in reporting the breadth of their work across health, anti-racism/discrimination, education, advocacy, and social domains. But activity monitoring on its own is limited: it records outputs rather than outcomes; busyness rather than contribution; and it places a substantial administrative burden on already under-resourced staff. Most critically, it tells us very little about whether the programme is achieving the changes it was designed to bring about. Meaningful evaluation of actual outcomes remains underdeveloped.
The “Catch-22” of community development
Ireland has had a strong community development infrastructure developed over the past forty years. According to the All-Ireland Endorsement Body for Community Work Education and Training (AIEB) community work/community development is a unique and distinct professional practice which is comprised of both a task and a process.[vi] The task is social change, to achieve equality, social justice and human rights, and the process is the application of principles of participation, empowerment and collective decision-making in a coordinated and structured way.
Like many community development projects in Ireland, the PHCTPs have been caught in a structural cycle of short-term funding to maintain their programmes of work with limited opportunities for the kind of long-term strategic thinking and development that sustained funding would support. This results in a situation where PHCTPs, like many community development projects, are not adequately positioned to evidence impact through current data collection methods, which weakens their case for funding, particularly the sustained, long-term funding that would support growth and development into the future. In essence, they are trapped in a cycle where insufficient data justifies insufficient funding.
This is further compounded where outcomes are not made explicit, or where there is a lack of consensus across stakeholder groups on what constitutes “tangible” results. The consequence is that under-resourced programmes cannot invest in better monitoring and evaluation approaches, and the status quo of narrow activity measurement persists. This dynamic actively undermines the communities these programmes are designed to support by placing unreasonable expectations on them to evidence change that they were not set up to support in the first instance, or that they were not adequately consulted on. Overall, this can undermine community efforts, demotivate staff, and polarise stakeholders.
The outset and context of the commissioning of the PHCTP evaluation by the Department of Health set a tone of accountability on the part of the PHCTP to evidence outcomes that are, in practice, co-produced across the HSE, social services and the PHCTPs. This raises a question of accountability: who should be responsible for evidencing what, particularly in programmes where outcomes are co-produced across the health system? PHCTPs cannot be held solely accountable for health outcomes that depend equally on the accessibility and responsiveness of primary care, social services, and wider supports. If the signposting and navigation work of a TCHW leads to a health service that is inaccessible or unresponsive, the failure does not sit with the PHCTPs.
This is not a problem unique to the PHCTPs. Researchers examining evaluation in community development more broadly have identified it as a persistent and structural issue. As Mc Ardle and Murray (2021) observe, evaluation in this field is complicated by the long-term and non-linear nature of community development work, the absence of suitable measurement tools, and the enduring tension between accountability to funders and meaningful learning for communities.[vii] At the core, there is often a misunderstanding or multiple, conflicting expectations of what community development is and does. The PHCTPs are a clear illustration of what happens when these conditions converge.
The challenge of monitoring and evaluating community development
Community development, with its associated principles of social justice, solidarity, equality and human rights; and an approach that involves participation, empowerment and collective action has been fundamental in identifying structural issues and required responses, thus contributing to, and creating conditions for, a fairer and just society. The impact of this work is not primarily measurable in discrete, time-bounded activities and outputs. Rather, it accumulates slowly, often in ways that are difficult to attribute to any single, isolated activity or moment. It is evidenced in the confidence built over years – for example in a family that navigated a health crisis because a trusted person was there and connected them to the right services at the right time, and in a community that can advocate for itself more than it had the opportunity or means to do a decade ago. Progress is also demonstrated in processes, such as the commitment to work collectively to address structural issues.
Conventional indicator-driven frameworks tend to capture what is easy to count rather than what is meaningful to measure. Mc Ardle and Murray (2021) draw a useful distinction between task outcomes (meaning what is achieved), and process outcomes (meaning how it is achieved and what changes along the way).[viii] For PHCTPs, both dimensions matter. The aim is improved Traveller health, but the process – trust, empowerment, the relationship between a TCHW and a family, challenging racism and discrimination – is not merely a means to an end. In many respects, it is the end itself. Reducing evaluation to a set of activity counts renders the process dimension invisible and, in doing so, renders the most important part of the work invisible too.
This connects to a broader question about the purpose of evaluation. As Mc Ardle and Murray (2021) note, evaluations are too often commissioned to satisfy funders rather than to generate learning for communities and services.[ix] Where this is the case, programmes may collect data that looks accountable without providing insight and learning into what is working, why, or for whom. It also potentially undermines and undersells the skills, expertise and knowledge required to undertake the work competently. O’Donnell et al (2026), writing on relational public services, raise a similar question: whose learning counts?[x] If monitoring systems are designed primarily around the information needs of commissioners and funders, they may tell us very little about what matters to Travellers, or what TCHWs understand about the change they are generating from the ground up.
What would better monitoring and evaluation look like?
Addressing this challenge requires a different approach to monitoring and evaluation and we propose that the following four principles are considered:
Proportionate:
Monitoring must be designed in proportion to the nature and scale of the programmes being assessed. This matters in two related ways. First, it is unreasonable to expect small teams, that are under-resourced and/or without parity of resources, to sustain complex data collection systems on top of frontline delivery. Second, and more fundamentally, the framework used must be suited to what is being measured. Using a standardised activity count to assess the impact of a community development programme is rather like using a thermometer to assess whether a patient feels heard: it captures one dimension of the situation while missing what matters most. As Motherway (2006, cited in Mc Ardle and Murray, 2021) observed, there is no simple, universal solution to measuring community development impact, and any attempt to impose one risks measuring what is easy rather than what is important.[xi] Relatedly, evaluation budgets and expectations about what can be measured and how, need to be proportionate to the resources allocated (e.g. commissioners should consider the costs of the evaluation as a percentage of the cost of the programme; and align expectations accordingly).
Meaningful:
Better monitoring must generate data that says something real about the contribution that a programme is making towards the social problems it was designed to address. This requires both qualitative and quantitative data, and space for learning. Participatory approaches offer a more appropriate toolkit for this kind of work. Methods such as Most Significant Change, Community-Based Participatory Research, Photovoice, Developmental Evaluation, Outcome Harvesting, Outcome Mapping and Ripple Effect Mapping have all been developed in line with participatory approaches. They can support researchers and evaluators to capture change from the lived perspective and should be more readily incorporated and commissioned as part of evaluations of health and social care services in Ireland.[xii] These approaches are methodologically robust and can be combined with quantitative data to explore causality and contribution. McNally (2026), writing on place-based and community-led approaches in the Irish public health context, argues that stories, collective [lived] experience, and community-defined accounts of progress are often indispensable in understanding how change occurs and why it endures.[xiii] The same principle applies to PHCTPs and to community development monitoring more broadly.
Long term commitment:
Community development does not necessarily produce results within annual reporting cycles and KPI deadlines. The work of social change takes time and possibly generations to come to fruition. Building a foundation of trust requires patience and a genuine commitment to partnership working over the long term. There is a need to challenge the expectations of funders and commissioners who – either implicitly or explicitly – expect social change to be evidenced quickly. This can be addressed if all stakeholders clearly articulate their expectations and agree on what is reasonable to evidence and when. Changes in health behaviour, community confidence, advocacy capacity, and systems-level change emerge over time and in ways often difficult to attribute to a specific intervention or funding period. Short-term evaluation frameworks, driven by short-term funding cycles, are structurally ill-suited to this kind of work (Mc Ardle and Murray, 2021).[xiv] Investing in meaningful monitoring of community development requires an imperative of being very honest about what can reasonably be expected in terms of short and long-term impact, and a commitment to sustained, long-term funding. Without this, even a well-designed evaluation framework cannot produce the evidence it promises.
Relational:
Perhaps the most important shift required is conceptual: recognising that relationships are not a soft feature of community development work but its core mechanism of impact. O’Donnell et al (2026) argue that in the context of public services more broadly, relationships are not an optional addition to real service delivery; they are the mechanism through which good outcomes are achieved. This insight has direct implications for how we design monitoring and evaluation frameworks for community development programmes, such as the PHCTPs.[xv] For example, if the relationship between a TCHW and a Traveller family is what makes health engagement possible, then a monitoring and evaluation framework that records only activities and contacts cannot see that relationship and risks missing its most important contributions.
This points toward collaborative outcome framework development. Rather than imposing top-down requirements, monitoring and evaluation for PHCTPs would involve TCHWs, Travellers, HSE staff, and policy makers working together to agree what outcomes matter, who is responsible for what, and how progress will be evidenced over time. This requires a sincere examination of how accountability is understood and distributed across the system.
Conclusion
The PHCTPs have been operating for over thirty years, building relationships and trust amongst families that have, and continue to face racism, discrimination, marginalisation and disadvantage. That they struggle to evidence their impact is not a reflection of their effectiveness. It reflects the inadequacy of the evaluation frameworks we have developed in our public health and social care system to understand and assess work of this kind. There is a lot to learn from this current weakness in our system.
This paper has argued that addressing the measurement challenge facing PHCTPs, and community development more broadly, requires approaches to measurement that are:
- proportionate to the nature and resources of the programmes involved;
- use meaningful, mixed methods including participatory approaches;
- allow for the long timescales over which community development impact accrues; and that
- place human relationships and partnership as a central mechanism for social change, rather than simply an incidental, “feel-good” or intangible byproduct of a programme or service.
The full PHCTP review sets out detailed findings and a series of recommendations for how this might be taken forward in practice, including the building blocks for a co-designed outcomes framework and guidance on the development of a theory of change for the PHCTP model. The case for change is well established. What is needed now is willingness from those designing and commissioning evaluation frameworks to move beyond activity monitoring, use the right tools, and involve the right people in deciding what counts as evidence of change.
The full report, Review of the Primary Health Care for Travellers Projects (PHCTPS), is available to download here:
References
All-Ireland Endorsement Body for Community Work Education and Training (AIEB) and Community Work Ireland (CWI) (2021) All-Ireland Standards for Community Work. Available at: https://www.aieb.ie/wp-content/uploads/2021/10/All-Ireland-Standards-for-Community-Work.pdf
Department of Children, Equality, Disability, Integration and Youth (2024) National Traveller and Roma Inclusion Strategy II 2024–2028. Dublin: Government of Ireland. Available at: https://www.gov.ie/en/publication/834f1-national-traveller-and-roma-inclusion-strategy-ii-2024-2028/
Donnelly, S. (2026) Review of the Primary Health Care for Travellers Projects (PHCTPs). Report prepared for the National Traveller Health Implementation Group. Available at: https://assets.gov.ie/static/documents/7c4a06ad/Review_of_the_Primary_Health_Care_for_Travellers_Projects_PHCTPS.pdf
Mc Ardle, O. and Murray, U. (2021) ‘Fit for measure? Evaluation in community development’, Community Development Journal, 56(3), pp. 432–448. Available at: https://doi.org/10.1093/cdj/bsaa005
McNally, L. (2026) Integrating Place-Based and Community-Led Approaches into Ireland’s Public Health System. Dublin: UCD Geary Institute for Public Policy. Available at: https://publicpolicy.ie/health/integrating-place-based-and-community-led-approaches-into-irelands-public-health-system/
Motherway, B (2006) The Role of Community Development in Tackling Poverty in Ireland. Dublin: Combat Poverty Agency.
O’Donnell, J., Lowe, T., Eichsteller, G. and O’Neill, A. (2026) Reframing Public Services as a Relational Practice. Dublin: UCD Geary Institute for Public Policy. Available at: https://publicpolicy.ie/papers/reframing-public-services-in-ireland-as-a-relational-practice/
York, M. and Stakem, M. (2015) Relationship and trust as a vehicle to improved health outcomes: A qualitative study of a Primary Health Care Programme for Travellers, Irish Journal of Applied Social Studies: Vol. 15: Iss. 1, Article 4. doi:10.21427/D7QT5W. Available at: https://arrow.tudublin.ie/ijass/vol15/iss1/4
Endnotes
[i] Throughout this paper, we distinguish between three related but distinct concepts. Monitoring refers to the ongoing systems for collecting activity data (typically counts of contacts, sessions, or information provided). Measurement describes the act of collecting and quantifying data. Evaluation refers to the systematic assessment of whether and how a programme is achieving its intended outcomes and contributing to the changes it was designed to bring about. The argument developed here is that PHCTPs are currently operating within monitoring systems that capture limited measurement data, while meaningful evaluation of their actual impact remains underdeveloped.
[ii] The report is available on the Department of Health website here. It was commissioned by the HSE on behalf of NTHIG and conducted and authored independently by Susie Donnelly in collaboration with a working group from HSE, Traveller organisations and Traveller Health Units.
[iii] The terms “community development” and “community work” are used interchangeably in the paper. The authors recognise community work as defined in the All-Ireland Standards for Community Work.
[iv] National Traveller and Roma Inclusion Strategy II (2024–2028)
[v] McNally, 2026; York and Stakem, 2015.
[vi] All-Ireland Endorsement Body for Community Work Education and Training (AIEB) and Community Work Ireland (CWI) (2021)
[vii] Mc Ardle and Murray (2021)
[viii] Mc Ardle and Murray (2021)
[ix] Mc Ardle and Murray (2021)
[x] O’Donnell et al (2026)
[xi] Motherway, 2006, cited in Mc Ardle and Murray, 2021)
[xii] Mc Ardle and Murray (2021)
[xiii] McNally (2026)
[xiv] Mc Ardle and Murray (2021)
[xv] O’Donnell et al (2026)