Evaluating Effectiveness of Institutional Health Partnerships: the EFFECt Tool
Today there are many programmes designed to improve public health globally, resulting in the creation of institutional health partnerships. This month we celebrate the release of the new version of ESTHER Alliance’s tool to measure and improve on its partnership approach.
Published: 15.09.2021, Author: Jeannie Wurz
“ESTHER focuses on providing support for people to work together and learn from each other,” says Judith Safford, head of programme for ESTHER Switzerland. “This is very different from the usual development project, which is about building hospitals or providing clean water.”
Monitoring and evaluation of public health programmes traditionally focuses on measuring activity. “A project either achieves its objectives or it doesn’t,” says ESTHER Ireland’s David Weakliam, who is Global Health Programme Director in Ireland’s Health Service Executive. What is more difficult to assess, Weakliam says, is the extent to which a partnership has contributed to a project.
Several factors contribute to successful outcomes, according to Weakliam. First, a project is typically time bound, whereas a partnership tends to take a more long-term approach. Next, the partnership should be based on sustainable change. “How do you embed positive change in the institutions you’re working in?” And finally, “What are the things that matter?” The EFFECt tool was developed to look at some of these different parameters.
Developing the tool
In 2012, when Ireland joined the ESTHER Alliance for Global Health Partnerships, there was agreement over the need for a way to evaluate healthcare partnerships. There were “lots of tools out there for evaluating projects, but there wasn’t really anything for evaluating partnerships,” says Weakliam.
After much discussion and strategizing among members of the ESTHER Alliance and other key players in global health, ESTHER chose Vicki Doyle and Ema Kelly of Capacity Development International to work on the development of a pilot tool in 2016.
“There were countless nights, weeks and months of myself and Ema poring over the literature, and thinking about what the tool could look like,” says Doyle. They based their tool on an extensive review of published and grey literature evaluating the approach and impact of Institutional Health Partnerships. What they discovered, and reported in a 2015 article authored by Kelly, Doyle, Weakliam and Yvonne Schönemann (ESTHER Germany), was that “evidence for the effectiveness of Institutional Health Partnerships is thin, both in terms of quantity and academic rigour”.
“The goal was to go beyond routine reporting of activities and outputs, and challenge Institutional Health Partnerships (IHP’s) to think about impact, reach, lasting benefit, sustainability and evaluation from the start,” says Doyle. “We found that many partnerships were doing great work, but the routine M&E (Monitoring and Evaluation) wasn’t really picking up on what it was they were doing.”
Testing the tool
In 2017 a first version of the tool was ready for pilot testing. It was named the ESTHER EFFECt Tool, standing for EFFective in Embedding Change tool.
Ahmed Razavi, a consultant in global public health for Public Health England, was closely involved in testing the tool with partners in Nigeria. “It was really invaluable to have all these senior leadership team members sitting across the table from one another and learning about the tool,” he says.
Razavi and his co-workers described the evaluation process in their 2021 publication Evaluating an institutional health partnership using the ESTHER EFFECt tool: A case study of an evaluation of the institutional health partnership between Nigeria CDC and Public Health England
“Actually measuring the value of the partnership is quite difficult to do,” says Razavi. “The EFFECt tool remains one of the only tools that actually looks into the quality of the partnership, which is really vital to ensuring that any sort of aid or development work is successful in its implementation.”
Using the tool
ESTHER Ireland used the first version of the EFFECt Tool to evaluate its partnership with the Ministry of Health in Mozambique, says David Weakliam. “What became apparent was that we wanted to do it and they didn’t necessarily want to do it.”
Ahmed Razavi stresses the importance of being mindful of the context. “Too often we have this patriarchal way of engaging with our southern partners, and we may misunderstand what IHPs look like and how they are perceived” at the other end, he says. As the partners use the Tool to work together, says Razavi, “recognising and adapting to the different ways of working and the different cultures and contexts that we live in, and adapting and delivering the Tool to your context, is vital.”
Ultimately, says David Weakliam, “a partnership is not just about achieving good results at the end of a project. It’s about embedding lasting and sustainable change.”
The revised ESTHER EFFECt tool and guidance for use are available in English and French on the Resources page under “Tools”.
The current article incorporates information presented at the ESTHER Alliance’s new webinar series, which started in April 2021. The first topic was “Evaluating Institutional Health Partnerships,” with speakers David Weakliam, Vicki Doyle, and Ahmed Razavi. You can follow this event here.