Towards more inclusive and feminist approaches in evaluation of HIV programming - transforming principles into practice

In 2020, Social Development Direct was contracted by UNAIDS Evaluation Office to conduct an evaluation of the Joint Programme’s work on addressing the linkages between HIV and VAWG – where violence can be an indirect and direct factor for increased HIV risk for women and girls, and violence can be an outcome of HIV status and disclosure.  Coinciding with the launch of the evaluation report, this learning brief shares reflections on how the evaluation took steps to meaningfully involve women living with HIV in the 29-member strong, all female, evaluation team.

The importance of meaningfully engaging women living with HIV/ AIDS is widely recognised in HIV programming, research and evaluation: articulated by the principles of Greater Involvement of People living with HIV/AIDS (GIPA), and Meaningful Involvement of  Women living with HIV/ AIDS (MIWA). The evaluation was designed with these principles at the centre, and adopted an intersectional approach to understanding Violence Against Women and Girls (VAWG) – as it sought to understand how the Joint Programme has addressed violence against women and girls in their diversity, including women living with HIV, women from key populations, and women who experience intersecting HIV and VAWG risk due to overlapping forms of discrimination and violence. In addition, the evaluation adopted feminist evaluation principles, which call for critical reflection on power and privilege in knowledge production, focus on structural gender inequality and social injustice, and is utilisation-focused. Among other things, these principles required the evaluation to i) be guided by the priorities of women and girls living with and affected by HIV and be accountable to them and their networks; and ii) acknowledge and address structural inequalities present in common evaluation practice.

The remainder of this blog explores how we embedded these principles in the evaluation design and approach, and what practical steps we had to take to transform these principles into practice. 

Ensuring meaningful involvement of women living with HIV

1. Creating an advisory group of women in their diversity. During the inception phase of the evaluation we worked with the Salamander Trust, the International Community of Women Living with HIV/AIDS (ICW) and the Athena Network to establish The Accountability and Advisory Group (TAAG), a group of women living with and affected by HIV. TAAG supported the development of the methodology and tools, and TAAG members in each of the nine countries worked as part of the data collection and analysis team.
2. Collaborating in country teams. A three-person team was established in each of the nine countries, made up of one TAAG member, a national consultant and a member of the core evaluation team. Having this strong national representation on the evaluation team made it easier to reach relevant stakeholders, including government actors and civil society, especially in the context of an evaluation conducted remotely due to COVID-19. Each national team analysed and validated the findings together to ensure key considerations were relevant and contextualised for each of the countries.
3. Reaching women and girls in their diversity. All the TAAG members were well connected with the national HIV networks and could draw on existing connections and relationships of trust within their communities. The TAAG members spoke with representatives of key community-led HIV networks and ran focus group discussions (FGDs) with women living with and affected by HIV, focusing on their experiences of UN activities on VAWG and HIV and on UN accountability to community-led organisations and women living with HIV.  This became even more valuable given the COVID-19 context which meant face-to-face meetings were not possible in most countries.

A TAAG member commented on the inclusive approach:

"The process is very good because it involved the people in the community that are usually left behind; such as transgender women, women who use drugs, and women engaged in sex work; it includes also all women living with HIV. Especially that the evaluation is to see how the UN Joint Team works on VAWG and HIV areas, where they don’t really include those groups. So I think what TAAG adds is the space for us to speak and show the issue."

Validating findings and developing recommendations

TAAG members played an absolutely essential role in supporting the analysis and sensemaking of the findings, coming together at crucial points during the process to review and feed into the findings and recommendations. A key recommendation from TAAG was on the importance of meaningfully involving women in their diversity, including trans women, in UN decision-making. One TAAG member said that:

"[United Nations Programme on HIV/AIDS] (UNAIDS) should work towards bringing all UN agencies together, to work in tandem with grassroot organisations to address the links between HIV and VAWG, for there to be a synchronisation of such programs. [...] UN agencies should work with grass root organisations and not only fund huge organisations – grassroot organisations perform better and are more hands on the ground compared to bigger organisations."

What practical steps enabled this approach?

The operationalisation of Greater Involvement of People Living with HIV and AIDS (GIPA), Meaning Involvement of Women Living with Aids (MIWA), and the feminist principles that guided the evaluation, required SDDirect to take a number of practical steps, some of which meant re-thinking the usual way of doing things and actively removing barriers that limit the meaningful participation of women living with HIV in evaluations.

1. Negotiating and re-defining conflict of interest clauses to ensure these did not exclude women who are part of HIV organisations and networks in the Global South, by affecting their chances to access funding and opportunities in the future
2. Ensuring women living with and affected by HIV were fairly remunerated for their work as TAAG members and national consultants
3. Ensuring women in their diversity who took part in interviews and FGDs were fairly compensated for their participation and inputs
4. Co-creating the space for women living with and affected by HIV to shape the evaluation methodology and process and well as the findings
5. Ensuring documents, training materials and sessions, group meetings and reports were translated into the main evaluation languages (English, French, Spanish, Russian, and Khmer) to facilitate meaningful participation

Lessons learned and ways forward

Our ambition to conduct an evaluation that would be more rooted in national realities, framed by the priorities and experiences of women living with and affected by HIV in those countries, required us to question assumptions and standard practices in the sector, as well as adapt SDDirect’s internal systems to align with our feminist principles and commitment to GIPA/MIWA – leading to more inclusive systems and practices that should be adopted beyond this evaluation. Key lessons from this experience include:

• All evaluations should be guided by members of communities. This evaluation included women activists as TAAG members, as well as national consultants in some countries. This ensured the evaluation addressed the issues of priority to community members, and also supported the ongoing accountability for the implementation of evaluation recommendations. Women who are active in the communities provide a wealth of knowledge about what works and what doesn't, and this should be valued beyond being interviewed as key informants.
• Proper compensation for women’s contributions is essential as it recognises the value of women’s knowledge, time and commitments, and can help remove barriers to participation when paid in advance and through flexible modalities.
• Internal systems need to be flexible. In addition to adapting the approach to how payments were made, SDDirect adjusted the contracting processes to ensure they were appropriate to the roles and the experience the women had, in particular adapting lengthy documents, ensuring critical elements were translated and Certificate of Insurance (COI) requirements were adapted.

Reflecting on the TAAG process, the 13 TAAG members not only contributed with their expertise and made it possible to reach women living with and affected by HIV that would otherwise not have been able to engage – the process also meant that women living with HIV built on existing -  and created new - connections with UNAIDS and other stakeholders, as well as with each other:

"Hopefully it [the evaluation] will be really listened to, not only by all the United Nations Joint Team but by the government in our country. The most important of all is that I can meet you all, the fabulous women from other countries. Despite the different languages."

In some cases, the discussions held with key informants led to immediate changes, such as new momentum to advocate on VAWG in relation to HIV, or new relationships between VAWG actors and networks of women living with HIV. In all cases, the involvement of the TAAG members in the evaluation will hopefully mean there is accountability for lasting change within the case study countries and globally.

To meaningfully promote community leadership, feminist and anti-racist approaches, level power differences, and embrace GIPA/MIWA, the concept of ‘independent evaluation’ needs to be revisited. Women who are active in their communities have a wealth of knowledge about what works and what doesn't, and evaluation practice should be centred around this, ensuring this knowledge and expertise is recompensed and recognised.