Evaluation of the Reproductive Health Supplies Coalition
Publication date: 2012
305968 01 01 /// 20 June 2012 Evaluation of the Reproductive Health Supplies Coalition Commissioned by the Bill & Melinda Gates Foundation in association with USAID, DFID, and UNFPA Don Lauro Adrienne Chattoe-Brown (HLSP) 30th June 2012 Final report 305968///01/01 30 June 2012 Evaluation of the Reproductive Health Supplies Coalition Many people gave a great deal of time and thought to help with and support this evaluation. Representatives of the four donors funding the RHSC, DFID, the Gates Foundation, UNFPA, and USAID, developed a solid SOW for us to build on and then met with us in person to provide input for our evaluation work plan. At a later stage, they joined in a conference call to help us sift through preliminary findings. The staff members of RHSC, and particularly the Director, were patient with our presence in their office, the many hours of interviews we conducted there, and our frequent follow-up requests for additional documents and information. The members of the RHSC, who we interviewed as key informants, were open and willing to participate, much like their contributions to the Coalition over the years. We can honestly say that we learned something new about the Coalition from each and every person interviewed. The 65 members who responded to the electronic survey sent to all member organizations also gave us new perspectives and additional insight on the RHSC. Throughout this evaluation, we received a great deal of support from a variety of people for several tasks: conduct background research, transcribe interviews, develop and administer the members’ survey, and organize and format this report. We thank all who have participated in this evaluation and trust that they will find value and use for what we have learned about the RHSC and what we recommend for its future. Acknowledgements 305968///01/01 30 June 2012 Evaluation of the Reproductive Health Supplies Coalition Acknowledgements i Content ii Acronyms 1 1. Executive Summary 2 1.1 Introduction ________________________________________________________________________ 2 1.2 Achievements: Global, Regional, and Country Level ________________________________________ 3 1.3 RHSC Structures and Processes _______________________________________________________ 6 1.4 Future Directions and Recommendations for the Coalition ____________________________________ 8 2. Introduction 10 2.1 Scope of Work ____________________________________________________________________ 10 2.2 Organization of the Report ___________________________________________________________ 10 2.3 Methodology ______________________________________________________________________ 11 3. What contributions has RHSC made and what impact has it had on improving commodity security at global, regional, and country levels? 12 3.1 Overview _________________________________________________________________________ 12 3.2 Performance against the monitoring and evaluation framework _______________________________ 13 3.3 Country uptake and results of RHSC-inspired initiatives _____________________________________ 13 3.4 Impact at regional level ______________________________________________________________ 16 3.5 Influence on the global health agenda __________________________________________________ 16 3.6 Contribution of the Coalition __________________________________________________________ 16 4. Over and above partner organizations’ own accomplishments, what is RHSC’s comparative advantage, added-value, and value-for-money? 18 4.1 Comparative advantage _____________________________________________________________ 18 4.2 Added value ______________________________________________________________________ 20 4.3 Value for money ___________________________________________________________________ 22 5. To what extent is RHSC’s role adequately reflected in its strategy, theory of change and M&E framework? 26 5.1 Current strategy ___________________________________________________________________ 26 5.2 Theory of change __________________________________________________________________ 27 5.3 Monitoring and evaluation framework ___________________________________________________ 27 6. In what ways and to what effect has the structure and operation of RHSC evolved, particularly since 2009? 31 6.1 2009 Terms of Reference ____________________________________________________________ 31 6.2 Operational Structure of the RHSC _____________________________________________________ 33 6.3 RHSC Flexibility and Adaptations ______________________________________________________ 34 6.4 Support for Structures and Operations __________________________________________________ 35 6.5 Future Secretariat Needs ____________________________________________________________ 36 7. What works well and what needs to be improved or changed about RHSC’s internal structure Content 305968///01/01 30 June 2012 Evaluation of the Reproductive Health Supplies Coalition and external operations? 37 7.1 Strengths within RHSC Internal Operations ______________________________________________ 37 7.2 Weaknesses within RHSC Internal Operations ____________________________________________ 39 7.3 External Operations ________________________________________________________________ 40 7.4 Appropriateness of RHSC Structures and Processes _______________________________________ 41 8. To what extent has RHSC been responsive to partner-led initiatives or proactive towards new opportunities? 43 8.1 Partner-Led Initiatives within RHSC ____________________________________________________ 43 8.2 A Proactive RHSC _________________________________________________________________ 45 8.3 Coalition Upsides/Downsides _________________________________________________________ 46 9. Does RHSC seem poised and/or have potential for improving and increasing contraceptive security in the future? 48 9.1 Country Level: Origin and Destination __________________________________________________ 48 9.2 The RHSC Membership _____________________________________________________________ 49 9.3 Breadth and Depth of the RHSC _______________________________________________________ 50 9.4 Poised but Not Fully Positioned _______________________________________________________ 52 10. Beyond serving as an umbrella for collaboration, what alternative models or approaches could and should the RHSC adopt that are needed in the coming years to advance RH commodity security globally, regionally, and at country level? 53 10.1 Strategic ways forward ______________________________________________________________ 53 10.2 Operational issues _________________________________________________________________ 57 Annex 1 Statement of Work 59 Annex 2 Bibliography 66 Annex 3 RHSC Evaluation Workplan 69 Annex 4 Answers to Key Questions from RHSC Scope of Work 87 Annex 5 RHSC Member Survey 2012 113 Annex 6 Evaluation Methodology and Data Analysis 142 Annex 7 Summary of results against the MEF 145 Annex 8 Outcome of CARhs engagement. 149 Annex 9 Country relevant Innovation Fund grants 150 Annex 10 Contributions of the Supplies Coalition to Key Initiatives 152 Annex 11 Added Value of the Supplies Coalition 156 Annex 12 Presentation to Executive Committee, June 25th 2012 158 Annex 13 Reproductive Health Supplies Coalition. Monitoring Report (2009-2011) 169 1 Evaluation of the Reproductive Health Supplies Coalition BMGF Bill and Melinda Gates Foundation CARhs Coordinated Assistance for Reproductive Health Supplies DFID UK Department for International Development FP/RH Family Planning and Reproductive Health GFATM The Global Fund to Fight AIDS, Tuberculosis and Malaria GHP Global Health Partnership GPRHCS Global Programme to Enhance Reproductive Health Commodity Security H2H HANDtoHAND Campaign HLF High Level Forum IAI Implanon Access Initiative IF Innovation Fund JSI John Snow Inc. LAC Latin America, Caribbean M&E Monitoring and Evaluation MEF Monitoring and Evaluation Framework MDAWG Market Development Approaches Working Group NGO Non-governmental Organisation PAHO Pan American Health Organization PGH Pledge Guarantee for Health RH Reproductive Health RHCS Reproductive Health Commodity Security RHI Reproductive Health Interchange RHSC Reproductive Health Supplies Coalition RMAWG Resource Mobilisation and Awareness Working Group SECONAF Sécurité Contraceptive en Afrique Francophone SOW Scope of Work SSWG Systems Strengthening Working Group UN United Nations UNFPA UN Population Fund USAID United States Agency for International Development WAHO West African Health Organization WHO World Health Organisation Acronyms 2 Evaluation of the Reproductive Health Supplies Coalition Julie Solo’s 2011 review captures how “no product, no program” moved from sideshow slogan to mainstream movement: “(in the late 1990s) a small group of committed and driven women…saw concern for supplies gradually slipping,…committed (themselves) to preventing that, and (formed) the Interim Working Group on Reproductive Health Commodity Security”. Reformulated in 2004 as the Reproductive Health Supply Coalition (RHSC or Coalition), 12 (mostly donor) organizations continued to coordinate work around family planning and reproductive health (FP/RH) commodity security. By 2006, with funding from the Bill and Melinda Gates Foundation, the RHSC had expanded to some 20 members, began supporting members from a Secretariat based in Brussels, organized into three Working Groups, and was overseen by an Executive Committee comprised of donor and other members. Presently, with this same structure basically intact, RHSC has grown to include 200 organizations and has originated and advanced a number of approaches to improve how FP/RH commodities are positioned, procured, and provided to those who wish to use them. Four major donors currently provide core support to the Coalition: the Gates Foundation; the United Kingdom’s Department for International Development (DFID); the United Nations Fund for Population Activities (UNFPA); and the United States Agency for International Development (USAID). These four donors commissioned this evaluation and identified two consultants, Don Lauro and Adrienne Chattoe-Brown, to conduct it. Carried out from March through June, 2012, this Evaluation Report presents the consultants’ major findings. As categorized in the Scope of Work (SOW) at Annex 1, the findings of this evaluation are summarized as follows: Achievement of the RHSC within family planning and reproductive health (FP/RH) commodity security have at global level been generally positive and in some areas impressive; regional-level accomplishments are promising but more recent; and country- level impact has to date been fairly weak. (The Coordinated Assistance for Reproductive health supplies (CARhs), a group that works effectively “to address short-term supply crises that periodically befall countries” is the exception that proves the point. Other advances at global level, such as AccessRH, should eventually have significant impact on commodities at country level) Structures and Processes that the RHSC has developed and supports are solid and reasonably productive; however, the composition and what has been accomplished through these structures are not fully or currently in line with the rapid membership growth that the Coalition has experienced. The strategic goals, principles and assumptions associated with the 2007-2015 Program Strategy need to be revisited to address current challenges, consider new opportunities, and move the Coalition forward. Future Directions for the Coalition are to realign structure as well as strategy to reflect current and future membership; continue progress for commodity supplies at global and regional levels; and become more directed and committed to having impact at country level. 1. Executive Summary 3 Evaluation of the Reproductive Health Supplies Coalition Following the SOW, the methodology that the evaluators used included key informant interviews (in all, over 60 representatives of Coalition members and staff were interviewed) together with reviews of RHSC documents and other relevant literature (listed at Annex 2). In addition, as discussed with donors and included in the Evaluation Work Plan (Annex 3), the evaluators added an electronic survey of all members (65 members responded) and observed/participated in a few Coalition activities, such as Working Group conference calls and meetings. With details covered in 19 key questions included in the SOW, this evaluation is organized around three main subject areas: Achievements; Structure/Processes; and Future Directions. As the RHSC donors agreed early in this process, within these three broad categories the evaluators reformulated the 19 questions into eight overarching questions. These overarching questions also provide the outline for the body of this Report (Sections 3 to 10), with the findings in relation to the 19 SOW questions provided in Appendix B. The remainder of this Executive Summary summarizes major findings within the three SOW core areas. The 2007-2015 Strategic Plan is a roadmap against which RHSC achievements may be assessed and measured. The RHSC Strategic Plan has three goals each encompassing three objectives, shown below as on the RHSC website: Goal One: Increase Resources Mobilize Additional Resources Enhance Use of Existing Resources Maximize Total Market Goal Two: Strengthen Systems Strengthening Functioning Systems Respond to System Failures Strengthen Response to Crises Goal Three: Add Value Foster Greater Harmonization Expand Knowledge Base Frame Global Efforts For the most part, Goals One and Two delineate what member organizations would accomplish on their own, collectively, and sometimes within Working Groups. However it is Goal Three that describes what the RHSC itself does: “Assure the added value of the Coalition as a productive and sustainable global partnership through efficiency, advocacy, and innovation”. Five years on, achievements of the RHSC and members have been considerable at the global level; somewhat productive and promising at the regional level, though recent and not necessarily replicable; and at country level both small scale and sporadic for the most part, particularly in relation to the many countries with high unmet need and corresponding weaknesses in supply systems to serve those needs. . In fairness, global efforts have been on RHSC’s agenda for well over five years while ability to reach out to countries is relatively recent. In addition, some global achievements have had impact in some countries. For example, through combined inputs of several donors and organizations, CARhs has alleviated stock-outs in several countries; the Pledge Guarantee for Health has effected price reduction for Implanon in Ethiopia; and advocacy for commodity security has made progress in countries such as Ghana and Uganda. 4 Evaluation of the Reproductive Health Supplies Coalition The prominence of FP/RH commodity security has clearly increased since the 2001 Istanbul Conference first brought international attention to this issue. From 2005 onwards, including within the 2011 Access for All Conference in Addis Ababa, the RHSC has been at the forefront and increasingly the face of this movement. The Coalition has raised the profile of FP/RH commodity security on the global health agenda. Illustrative of advances are members’ engagement with the UN Commission on Life Saving Commodities for Women and Children to include specific contraceptives and maternal health supplies, and the HandtoHand (H2H) Campaign which has brought increased recognition to the importance of contraceptives as a strategy to reduce maternal deaths. H2H was also a forerunner to and underlays the rationale and impetus of the upcoming FP Summit. To promote global level advances, the Coalition has nurtured and its members have developed a number of tools that already have and will continue to increase and improve FP/RH commodity security for years to come. Among these are approaches that facilitate procuring large amounts of contraceptives more efficiently, resulting in cost and time savings, and sometimes lower product prices: AccessRH to electronically receive and process orders and RHInterchange to track procurements are landmark developments widely used. AccessRH led to saving an average 10.3 weeks in wait-time in 2011 The Coordinated Assistance for Reproductive Health Supplies Group (CARhs) reacts to and remedies stock-outs which threaten supply flows in 22 countries at risk. It also proved effective at addressing overstocks and resolving situations where stock levels are below minimum standards but not stocked out Total Market Initiative, based on analysis of market segments for contraceptives, has been successfully applied if not fully realized in two countries, Honduras and Madagascar FP/RH manufacturers appreciate the neutral space that the Secretariat occupies. This has led to better understanding of how market dynamics work within global supply chains and joint ownership of commodity security by procurers and suppliers. Better communication has resulted in significant price reductions in Implanon, Jadelle, and the female condom of at least $22m. Advocacy by local civil society organizations of host country governments has advanced local capacities for contraceptive procurement and commodity security in several countries, most notably in Uganda and Ghana The Pledge Guarantee for Health (PGH), although slower to take off than originally predicted, shows promise through its contributions to brokering the price reductions in the procurement of Implanon, which has to date been particularly beneficial for the substantial procurements made by the Government of Ethiopia Product pre-qualification, moving forward steadily if somewhat slowly, is making both quality and availability the hallmarks for future procurements of FP/RH commodities The three RHSC Working Groups, and to a lesser extent the more focused Caucus on New and Underutilised Methods and Maternal Health Task Force, have been directly involved in and directly contributed to most of these global level accomplishments. It is important to sustain such global level efforts and in particular to be attentive to promoting and pursuing through-lines to country-level impact. In this regard it would be useful for the Secretariat and RHSC structures to encourage coordinated and consolidated action by members working at country level. For example, incentives could be built into the successor to the Innovation Fund for awards able to demonstrate collaboration and/or impact at country level. Similarly, the Secretariat could more thoroughly map the country presence of members, local as well as international organizations. Even established and largely successful global partnerships such as CARhs, which depend upon USAID, UNFPA, and other international donors for information 5 Evaluation of the Reproductive Health Supplies Coalition about supply shortfalls, could benefit from fuller knowledge about other member organizations working within particular countries. In recent years, two regional forums have joined the Coalition, the Latin America and Caribbean (LAC) Forum and the Sécurité Contraceptive en Afrique Francophone (SECONAF) for SubSaharan African French-Speaking Countries. For different reasons, each of these forums offers distinct advantages to their respective regions: The LAC Forum provides a vehicle for the region to remain globally engaged with FP/RH as donor support for programs and commodities in the countries winds down; SECONAF provides a new avenue for engaging with a large number of French-speaking African countries where FP/RH lags far behind. Regional memberships are recent within the Coalition and not necessarily a blueprint for reaching out to countries in the future. Though too soon to have demonstrated substantive achievements, progress made within these two forums, and particularly how advances are made, will be important in the coming years. The LAC Forum, for example, to serve needs within its region, is replicating the Working Group structure and the Innovation Fund mechanism of the RHSC. SECONAF in contrast is about to receive funding through a private foundation grant to RHSC to engage a regional consultant who will interface with countries in its region. Countries at different stages of development for FP/RH may well require different approaches. The 2009 Terms of Reference states that “the RHSC commits itself to achieving a sustained supply of affordable, quality reproductive health supplies in low- and middle-income countries”. Having such impact does not mean that the RHSC operates fully within countries to advance FP/RH supplies and their availability. Rather it is the combined work of Coalition members that will ultimately produce progress at country level. The guiding principle that “increased country ownership is fundamental for reproductive health supplies” directs the RHSC toward involving and engaging countries in setting the agenda. This and other RHSC guiding principles derive from earlier work by the High Level Forum on best practices for global health partnerships. To date, progress at country level has been the least of RHSC accomplishments. The Resource Mobilisation and Awareness Working Group (RMAWG) has been the most directly involved at country level. This has largely occurred as a result of separately funded advocacy projects run by RMAWG members, and these results have been relatively small scale, though perhaps significant within a few countries such as Uganda and Ghana. The Market Development Approaches Working Group (MDAWG) also initiated in 2010 country level applications of the Total Market Initiative. However, selection of Honduras and Madagascar proved politically untimely, with results that emerged towards public sector understanding of private sector potential largely lost. More broadly, though more sporadically, the RHSC has engaged and made contributions within some countries. Its work in Ethiopia in particular, where price reductions for Implanon are resulting in millions in cost savings and increased access for hundreds of thousands of women, is a notable achievement. Instrumental in this occurring however was that the RHSC held its 2011 Annual Meeting in that country. Though annual meetings cannot be held to impact all countries, the principle and practice of price reductions in relation to volume may now be broadly replicated to have far-reaching impact. 6 Evaluation of the Reproductive Health Supplies Coalition The core structures of Executive Committee, Secretariat, and three Working Groups are all solid, even stolid, mechanisms through which the business of the RHSC is carried out. The Executive Committee, particularly through leadership of a succession of strong chairpersons, has encouraged and presided over vast expansion of Coalition membership. The Executive Committee has also facilitated connecting RHSC with manufacturers for discussions about commodity security and supported including them as members of the RHSC. Most recently, the Executive Committee Chair provided impetus and inspiration for development and launch by the Coalition of the HandtoHand (H2H) Campaign. However, some seven years since initial formulation, the Executive Committee would benefit from being refreshed and reconfigured to more faithfully mirror the current membership of the RHSC. In addition, some greater depth of expertise and independent voices on the Executive Committee could increase capacity for providing long range oversight. For example, one or two Executive Committee seats could be set aside for notable experts, unaffiliated with member organizations, to further enrich discussions and perspectives on the RHSC. The Secretariat has faithfully and with great appreciation supported Coalition members. In the person of the RHSC Director, the Coalition has an effective facilitator, a nimble networker, and a leader who has worked well with the Executive Committee, including a succession of different chairpersons. He and the Secretariat have served the membership well by interfacing with those interested in FP/RH commodity security, including the growing number of new members. However, as RHSC membership has increased, new structures added, and the range of direct responsibilities increased, the Secretariat has become increasingly stretched. Resulting from some staff attrition as well as substantive expansion, some increase in technical expertise and experience at Secretariat level is indicated. Depending on new or expanded directions included within a revised and updated RHSC Program Strategy, additions to the Secretariat in terms of policy expertise, procurement proficiency, and country- level program experience should be considered. Working Groups and other RHSC structures, such as the Caucus for New and Underused Methods, the Task Force for Maternal Health Supplies, and the two regional forums, depend entirely on members’ voluntary participation, with ongoing and sometimes significant support provided by one of the Secretariat staff. The work that occurs within these structures embodies this “coalition of the willing” and exemplifies an organization that engages, as the Director states, in “the logic of leading from behind”. Most of the tools and approaches to advance FP/RH commodity security occur or have emerged from within these structures and this mode of operation. However, only relatively few of the many members are and can be actively engaged in these structures. Except within the forums, most new members have not in fact found ways to join with the long-term members who remain the stalwarts within these structures. That is a major challenge for the future – how to adapt current structures or re- structure so that there is wider opportunity for fuller engagement, especially by new members representing different constituencies. With such large increases in members having already occurred, the Working Groups and other structures have not correspondingly extended their reach. These RHSC structures thus risk becoming more an elite sub-culture of founding members than a vital and vibrant part of expanding interests and possibilities. 7 Evaluation of the Reproductive Health Supplies Coalition The comparative advantage of the Coalition lies in its neutrality, its reach, and its resulting power as a convener. With growth, the RHSC has greatly expanded the diversity of its membership, horizontally (from governments to commercial firms) as well as vertically (from country-level NGOs to large international donors). The commitment of its members and its small, efficient Secretariat are also assets. Its limited spending power has also given it strength as this has helped keep the Coalition a neutral space. However with expansion the Coalition is outgrowing the principle of sole reliance on volunteerism. The 2007-2015 Strategic Plan had anticipated adding value in several areas. These include providing a forum for technical collaboration, bringing together a ‘brain trust’ of expertise and experience, and promoting effective and efficient use of limited resources. Members are also able to identify a broad range of global, regional and country level efforts to improve commodity security where the Coalition has added value. With specific reference to Goal Three, the Strategic Plan envisioned that participation in the Coalition would add value in eight ways. As summarized in the evaluators’ assessments against these expectations in Table 1 below these have been partially realized: Table 1 Expected added value of participation in the Coalition Expected added value Assessment Providing a forum for technical collaboration and problem-solving. Demonstrated from the outset. Increasingly being used as such by a wider variety of members. Bringing together a “brain trust” of expertise, experience, and learning relevant to finding solutions to RH supply security. Partially demonstrated. Wide and growing membership from different constituents, with effective communications between many of them. However country level members not sufficiently engaged as a resource. Promoting the efficient and effective use of existing, limited resources by improved coordination and harmonization of RH supply programs. Some progress e.g. through CARhs and RHSC engagement in a variety of initiatives, and coordination between partners improved. But greater harmonization not yet evident. Bringing about change through unity of purpose and action. Less emphasis on this in past, but growing. There may be potential for capitalizing on this more. Increasing global awareness of RH supply issues and strengthening advocacy and influence globally and in countries through shared objectives and messages. Demonstrated at global level but less evident at country level. Promoting innovation and collaboration among the public, private, and commercial sectors to develop markets for RH supplies. Not yet demonstrated, but new relationships have been formed which could lead to this. Initial work has been done on total market approaches. Members joining together to seek funds for innovative ideas, research, advocacy, and best practices. Some Innovation Fund awards demonstrate collaborative effort, but there is potential for more. Several RMAWG members are joining together to secure funds to continue advocacy at country and other levels. Maximizing the impact of investments in RH supply security by leveraging the investments of other participating organizations. Demonstrated. e.g. increased funding for RH Supplies from GFATM. Good leveraging from members. It is difficult to assess value for money of the Coalition as a whole when none of its major initiatives (AccessRH, RHInterchange, CARhs, etc,) have themselves been completely evaluated. It is possible to identify however that the Secretariat is run efficiently and cost effectively, and as described above is essential to the operations of the Coalition. The Coalition has demonstrated good leveraging power to attract resources to RH commodity security initiatives. Including DFID’s recent support of nearly $16 million for CARhs, some $45 million in new funding has been leveraged since 2007. Coalition efforts have also led to 8 Evaluation of the Reproductive Health Supplies Coalition significant cost savings in the price of commodities of at least $22m. Contrasted with the $7.3m allocated by donors to the Secretariat for core costs and the Innovation Fund, generation of such savings alone would seem to be a particularly good return on their investment. The 2007-2015 Strategic Plan has generally served the Coalition well. However, the expanded membership, current conditions, and future possibilities suggest that it should be revisited in the near future. The goals are still highly relevant for the Coalition but a major revision of the monitoring and evaluation framework is needed. As it currently stands it is too wide, does not fully capture what the Coalition has or could hope to achieve, and is not used as a management tool or to promote accountability. Development of an overt theory of change, currently absent from though implied within the strategy, would facilitate development of a more useful and relevant Monitoring and Evaluation Framework (MEF). With membership likely to continue increasing, the future of the RHSC depends on three interrelated factors: Continue member commitment, widen involvement to newer constituencies, and sustain the sense of ownership that have been core to success within the RHSC Renew and moderately increase funding to support the RHSC, particularly the value- added contributions of the Secretariat, including modest increases in staffing and a reoriented re-granting mechanism (analogous to the Innovation Fund, but with modifications) to support activities and projects generated by members Refocus and revise the Strategic Plan to direct the Coalition towards having impact further along the supply chain, notably all the way to country-level to reach and ultimately serve more who wish to use RH commodities . To support movement towards the above, the evaluation team recommends the following principles and practices for the RHSC: Retain the core focus on RH commodities Maintain global as well as local focus on supply chain improvement Promote greater ownership of the Coalition by country-partners: government, civil society, and other country-level organizations Retain but supplement the partner-centric funding model by renewing, renaming, and reorienting the Innovation Fund and supplementing operations of RHSC structures Revise and update the Strategic Plan and support its implementation by adding staff with requisite experience and expertise to the Secretariat Continue selective proactive leadership, including taking clear public positions to move the FP/RH agenda forward The Coalition builds on a productive past derived from a straightforward central aim: move FP/RH commodities and supply chains, often afterthoughts, to the forefront. Though awareness and prominence about the importance of FP/RH supplies and products has increased since the Coalition has been in place, the RHSC should keep its core focus on commodity security. From this centre, however, the Coalition can do more to ensure that country members, especially those in countries where serious supply problems persist, are fully involved in finding solutions. As the RHSC maintains global and local focus on supply chains, reliance on partners and their voluntary contributions may also need greater supplementation and support. To this end, the evaluators suggest that the RHSC would put additional funding to good use. This should include not only some staff additions at 9 Evaluation of the Reproductive Health Supplies Coalition Secretariat but also renewed (and increased) support for a new Fund to support Working Group generated and country-inspired initiatives. The latter should provide some needed adjustments to support greater Global South participation and catalyze problem-directed solutions further along the supply chain. The current partnership with the Global Programme of the UNFPA as well as future arrangements to come out of the Family Planning Summit may both be able to make this happen. With membership having grown rapidly over the last few years and likely to continue doing so in the future, the RHSC is positioned and poised to continue contributing to RH commodity security. As in the past, it will do this by both adding to this field of knowledge and applying practical tools and approaches to improve systems at various levels. However, the Secretariat in particular does not currently have the bandwidth or depth among staff to do so. Furthermore, the RHSC Strategic Plan is somewhat outmoded and out-of-date. Unlike its broadened membership base, the current Strategic Plan does not contain cohesive pathways towards moving effectively and more fully along the supply chain. As a membership organization, the Coalition needs to identify means to involve members within this fundamental reformulation. It is neither too late nor too soon to begin doing so at the next Annual Meeting. An immediate challenge for the Coalition is to secure future funding even as it recasts itself for more involvement of members and moves toward fuller country level engagement. The evaluators have found sufficient productivity and promise within the RHSC to support application for continued funding. Leadership within RHSC remains strong, the Secretariat solid and well appreciated, and members increasingly well represent FP/RH and the collective needed to move forward. Structures and operations to date provide a solid base of experience upon which to improve. The future – needs as well as opportunities – make the Coalition a best case investment for going the next and last miles to make FP/RH available for all who wish to use them. 10 Evaluation of the Reproductive Health Supplies Coalition Among human endeavours, our highest aspirations and greatest potential are in working together towards a common goal. Collaboration among member organizations to ensure contraceptive and reproductive health commodity security is the founding principle of the Reproductive Health Supplies Coalition (RHSC or the Coalition). This external evaluation focuses on how well and with what results the RHSC has operated. As articulated in its 2007-2015 Strategic Plan, the RHSC vision is that “All people in (low and middle-income) LMI countries are able to access and use affordable, high-quality supplies, including a broad choice of contraceptives, to ensure their better reproductive health”. As specified in its 2009 Terms of Reference, “achieving (this) vision requires a multi-stakeholder approach to improve RH behaviours. RHSC’s membership includes multilateral organizations, bilateral and private foundation donors, low- and moderate-income country governments, civil society, intergovernmental organizations and NGOs, and the private sector”. The 2007-2015 Strategic Plan of the RHSC articulates three strategic goals: 1. Increase the availability, predictability, and sustainability of financing for RH supplies 2. Strengthen the capacity of health systems to deliver RH supplies in a sustainable manner 3. Assure the added value of the Coalition as a productive and sustainable global partnership through support for efficiency, advocacy, and innovation The purpose of this external evaluation is to assess what the Coalition has achieved more than midway through its Strategic Plan. To determine the effectiveness, efficiency, results, and impact of the Coalition and its partnership with nearly 200 member organizations, this evaluation addressed three core objectives from the Scope of Work (SOW): Achievement: Assess how well has the RHSC met its objectives and made an impact on Reproductive Health commodity security Structures/Processes: Assess how well have RHSC structures and processes worked towards achievement of RHSC objectives Future Directions: Recommend how should the RHSC's role evolve in light of the current and emerging environment, including measures to improve the efficiency and effectiveness of the RHSC To guide this evaluation, the SOW also included an extensive set of 19 additional questions (and sub-questions) aligned with these three evaluation objectives. (The complete SOW is at Annex 1.) During a mid-March planning meeting, the external evaluators and the donors supporting RHSC reviewed, discussed, and prioritized these SOW questions. Based on this, the evaluators subsequently summarized and included eight overarching questions in the Evaluation Work Plan (in Annex 3). The body of this report is organized around the eight over-arching questions. Evaluator findings in relation to the subset of the 19 key questions in the SOW are in Annex 4. The Executive Summary recapitulates overall findings in relation to the three objectives of this evaluation: Achievement; Structure/Processes; and Future Directions. 2. Introduction 11 Evaluation of the Reproductive Health Supplies Coalition Staff from the four major donors providing core costs for the RHSC and its Secretariat developed the SOW for this evaluation. This included a methodology using two principal approaches: a review of relevant literature and in-depth interviews with key informants. During a planning meeting between these donors and the evaluators, on March 15 in London, an electronic survey of RHSC members was added (Annex 5). Also, as discussed in this meeting and included in their March 20th work plan, the evaluators added a complementary approach: to a limited extent and within the evaluation timeframe, participation in and observation of RHSC operations and events. This four-part methodology and analytic approaches are detailed in Annex 6. 12 Evaluation of the Reproductive Health Supplies Coalition The RHSC has had demonstrable impact on commodity security at global level, made some promising progress regionally, and to date had some, but relatively less, impact at country level. At global level the tools nurtured by the Coalition, the Pledge Guarantee for Health (PGH), AccessRH, the RHInterchange and the Coordinated Assistance for Reproductive Health Supplies Group (CARhs), have demonstrably improved coordination, availability of information, prices for commodities, and effective use of donor money. The Coalition has also influenced the global health agenda through the HandtoHand (H2H) Campaign by increasing recognition of the importance of contraceptives as a strategy to reduce maternal deaths. The Coalition has contributed to ensuring that family planning commodities are included in the UN Commission on Life Saving Commodities for Women and Children. The Coalition has also improved commodity security by facilitating coordination between the main donors engaged in commodity security. The RHSC provides a forum for a wide range of stakeholders to come work together on commodity security, an essential role in such a diversely populated and complex field. It has in addition encouraged the participation of manufacturers who have traditionally been ‘outside the fold’ of commodity security partners. (This is discussed further under Section 4.1 on Comparative Advantage and Section 4.2 on Added Value.) At regional level progress has been made by encompassing the LAC forum within the Coalition and supporting its application of RHSC structures. However, the Francophone forum, SECONAF, has only recently joined and is too new to have shown results. CARhs and the PGH have directly improved the availability of commodities at country level. AccessRH and the RHInterchange are also likely to be improving availability, although this is difficult to prove. The HandtoHand Campaign has probably not improved country level commodity security directly – it is too new and its through-line to countries is not clear. While engagement by people from developing countries in the work of the RHSC has been limited, the Coalition has opened the door to dialogue by securing the membership of representatives of a number of governments: China, Ethiopia, Ghana, Mauritania, Nicaragua, Nigeria, Senegal, Tanzania, Uganda and Zambia. Additional support for the work of the Coalition via Project RMA and Advance Family Planning has encouraged advocacy of governments by civil society and built some local capacity in advocacy for commodity security. The contribution of the Coalition to these efforts has been through a variety of means. These include generating ideas, widening engagement of stakeholders in initiatives, supplying seed- corn funding, facilitating implementation, encouraging technical rigor, and encouraging the use of Coalition-associated tools. 3. What contributions has RHSC made and what impact has it had on improving commodity security at global, regional, and country levels? 13 Evaluation of the Reproductive Health Supplies Coalition This section reviews the performance of the RHSC as measured by the monitoring and evaluation framework (MEF). Section 5 discusses the strengths and weaknesses of the MEF. A summary table attempts to systematically assess how well the RHSC has performed (see the table in Annex 7 and its far right column, ‘progress’) but the absence of targets makes this subjective. The MEF does not currently capture indicators for performance at regional level. (The most recent monitoring report provided by the Secretariat is in Annex 13.) At global level there has been an increase in donor funding for contraceptives worldwide and an increase in the level of funding allocated to RH supplies by The Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM). However, contributions from both have decreased recently largely as a result of the global economic turndown including the suspension of the GFATM’s 2011 funding round. However four new donors have entered the commodity security scene (as measured by new donors to the Global Programme to Enhance Reproductive Health Commodity Security (GPRHCS)). Volatility of donor funding is being addressed by the Pledge Guarantee for Health, which has so far channelled $12m for implants in Ethiopia. AccessRH is now established within UNFPA and has dealt with $32m worth of orders. Despite current constraints – only condoms on stock and full payment required in advance for orders – AccessRH has already had orders totalling $32m. At the same time, the pre-qualification initiative is endeavouring to improve quality of available products and has pre-qualified 33 more products since 2007. These include seven contraceptives of which only two are lower-priced generic products. Much accomplished, much more remains to be done. The report against the MEF shows that all focus countries1 now have commodity security coordination committees established; more countries are accessing GFATM funds to support RH commodities; expenditure of national budgets for contraceptives has improved in many places, with many countries spending 100% or more of their allocated budgets; and some countries are starting to be aware of or use the tools available to them e.g., AccessRH, PGH, and RHI. Donor engagement at country level through the CARhs group has successfully averted stock-outs. Nevertheless, stock-outs across the 14 focus countries still show very mixed results: some countries are getting worse; substantive changes in commodity security through adopting a total market approach is not yet evident; and only one country of the four currently measurable shows a growth in the non-public sector share for key contraceptive commodities. The RHSC has engaged with commodity security initiatives in a variety of different ways. Some it has funded directly, for example through the Innovation Fund. Some the RHSC has facilitated through opportunities for networking and technical exchange among partners. The RHSC contribution to these initiatives is discussed below in section 3.6. The evaluation team assesses the following initiatives as those most closely associated with the RHSC and the most significant for countries, either in terms of proven or potential impact: _________________________ 1 14 focus countries (Burkina Faso, Ethiopia, Ghana, Mozambique, Rwanda, Tanzania, Uganda, Guatemala, Honduras, Mexico, Nicaragua, Bangladesh, Mongolia, and Nepal) were agreed by the RHSC on the basis that they are or were receiving targets support by RHI, Project RMA, and / or UNFPA’s GPRHCS. 14 Evaluation of the Reproductive Health Supplies Coalition The Coordinated Assistance for Reproductive Health Supplies (CARhs, formerly “Countries- At-Risk”) Group was established in December 2004. The CARhs brings together key commodity suppliers to address short-term periodic supply crises at country level. The group currently reviews data on 22 countries monthly. The table in Annex 8 shows the outcome of CARhs engagement from Oct 2010 – Sept 2011 The evidence suggests that CARhs is most effective at addressing overstocks (by delaying or cancelling future shipments), but is also effective at resolving situations where stocks are below minimum levels but not stocked out. The 2012 monitoring report (see Annex 13) reports that CARhs “saved money by arranging the transfer of more than US$2 million in excess commodities from Rwanda to Burkina Faso. It played a pivotal role in clearing millions of dollars’ worth of RH supplies stuck in port in Nigeria. In addition, it assisted with the procurement of six million condoms for a new military HIV/AIDS prevention program in Burundi.2” From the outset CARhs has impacted positively on commodity security at country level. Donors and implementation partners have worked together more closely with a systematic approach to address problems in which they all have a stake. Country participation has however been limited. Some stakeholders maintain that civil society and other country-based members could be part of an early alert system for potential stockouts. This deeper systemic approach to addressing stockouts has not yet been taken forward. The Pledge Guarantee for Health (PGH) is a financial tool that enables grant recipients (i.e. countries) to obtain short-term, low-cost commercial credit on the basis of pending aid commitments. This enables countries to buy donor funded commodities as needed rather than waiting for donor commitments to be realised. PGH was launched in October 2009 with the intention of improving the predictability and effectiveness of donor aid for RH commodity procurement. It is also now marketed as potentially able to reduce prices of commodities when used for large scale procurements, thereby enabling committed resources to go further. PGH operates within the United Nations Foundation. The PGH has taken longer to get underway than many had originally predicted. . High numbers of transactions were expected within a short timeframe. However, PGH has yielded only two successful executions so far. One of these was to support malaria bed nets in Zambia. However in the other case, the PGH contributed to significant price reductions in the sale by Merck of $12m worth of Implanon contraceptive implants to Ethiopia, reducing the cost per unit from $18 to $16.5. Lacking ongoing support for operating costs, the future of PGH is highly uncertain. This is unfortunate as the PGH has the potential to significantly impact commodity security at country level. AccessRH was launched in 2010 at UNFPA with design and financing support from multiple Coalition members. The purpose is to reduce volatility in supply resources though high- volume procurement, warehousing of commodities for rapid delivery, and transparent information on pricing and product costs. _________________________ 2 RHSC Monitoring Report (2009-2011), p23 15 Evaluation of the Reproductive Health Supplies Coalition Between Jan 1st 2011 and Mar 31st 2012 the value of condom orders (the only commodity currently held in stock by AccessRH) has exceeded $32m. Of this, $4.5m has been from UNFPA country offices (i.e. the captive market for the service which could be serving either governments or local organisations) and the remaining $27.5m has come from third party clients3. In comparison to previous years before AccessRH was set up, the number of external clients in 2011 has increased, with a larger proportion of these being new clients (74%). This implies a country level demand for this service and growing country reach. Shipping condoms from stock rather than production has led to a 10.3 week wait time saving in the course of 20114. Further improvement may occur when the plan to share forecasts for other commodities with manufacturers comes online to help them deliver on schedule and improve estimations of shipping dates. The Reproductive Health Interchange (RHI) provides open access to up-to-date, harmonized data on approximately $1 billion worth of shipments of contraceptive supplies for over 140 countries. In 2011 RHI migrated from John Snow Inc. (JSI) to UNFPA becoming part of AccessRH. The intention is to improve the quality and availability of data needed for planning, ordering and managing procurement. At the end of 2011 there were 632 active users registered, a substantial increase from 304 in 2009. Data from RHI is fed into the CARhs process and all of the Coalition’s 14 focus countries are registered users (which enables them to use the system for tracking shipment data). A total of 92 different countries have registered as users. User logins, an indication of the use of detailed reports, have ranged from about 22 to 35 per week during 2011, an increase on previous years implying regular use of the RHI. Ethiopia, Pakistan, Uganda, India, Kenya and Zambia all appear to be frequent users. The Hand to Hand Campaign was launched in 2010 with the intention of mobilising partners to achieve 100 million new modern contraceptive users in the next five years. Members are encouraged to support the campaign by increasing resources and service provision, making more contraceptives available at affordable prices, improving systems and carrying out international and national advocacy efforts. The Campaign has been unusual for the Coalition in that it relies primarily on the efforts of the Secretariat for implementation rather than partners. The Campaign, if successful, could have a major impact on countries. It was not possible within the scope of this evaluation however to assess the extent to which commitments to the Campaign represent additional efforts by the partners or whether the Campaign is just re-framing or perhaps refocusing existing efforts Some of the initiatives supported by the Innovation Fund have been intended to have some impact at country level. The table in Annex 9 summarizes results for those initiatives that have been completed. _________________________ 3 In 2011 these external clients consisted of 28 Ministries of Health or other public sector agencies (e.g. Social Security Institute, Ministry of Population etc.), 10 NGOs, 13 UNDP offices and 3 other UN Agency offices. 4 Source: AccessRH slides from UNFPA 16 Evaluation of the Reproductive Health Supplies Coalition The MEF does not capture results at regional level. However the RHSC Director reported that the LAC Forum has built a community of interest around commodity security; brought the Inter-American Development Bank in as a new donor in this area, as well as attracted funds from UNFPA and USAID; brought Latin America into the wider RHCS community; and created a link between the global community and countries in the region (viz. it helped solve a problem between a global manufacturer and a service provider NGO in the region). The francophone forum SECONAF is too new to have had an impact on regional commodity security. However, initial work supported by an Innovation Fund grant to Equilibres & Populations, has sparked interest among a range of stakeholders, notably including the Hewlett Foundation and the French government, and encouraged some initial advocacy efforts by civil society. The structure and operations of the RHSC are such that it is difficult to tease out the exact contribution of the Coalition as a whole, rather than the influence of individual members. However there are examples where there has been demonstrable added value of the Coalition to the global health agenda. Members of the RHSC working together helped to get family planning commodities included in the UN Commission on Life Saving commodities for Women and Children. The RHSC then contributed directly to the work of the Commission by advising on the selection of commodities to be included and carrying out data collection and analysis. The RHSC has also helped to bring provision of commodities into the maternal health debate by initiating the Hand to Hand Campaign. Part of the rationale for its development was to rally the support of the family planning community around the maternal health agenda, in particular around “Every Women/Every Child”. The H2H Campaign can be seen as the forerunner of the July 2012 Family Planning Summit which represents a widening realization of the importance of contraceptives to achieving MDG 5 and indirectly MDG 4. Generally the RHSC is helping to move family planning into a more central position; previously family planning had been side-lined by or subsumed into wider agendas, such as maternal health and HIV/AIDS, and had consequently lost focus. RHSC has also helped reduce the distance between the family planning and maternal health communities. The RHSC has also helped to demonstrate the potential contribution of manufacturers to the wider development agenda, positioning them as partners in family planning rather than adversaries, as has been the case in some global health initiatives. This has relevance to the work of UNITAID, the Clinton Health Access initiative, and bilateral donors working to improve access to medicines. From a regional perspective, feedback from interviewees in the LAC forum suggests that engagement of the Coalition in this region has helped to retain some global level attention on a region which is no longer a recipient of significant amounts of aid monies, but which still has significant commodity security issues for a large population. The contribution of the Coalition to these efforts has been through a variety of means. These include: 17 Evaluation of the Reproductive Health Supplies Coalition Generating ideas – by members of the Executive Committee and the Secretariat, by partners in Working Groups, or by members attending other Coalition meetings Widening engagement of stakeholders in initiatives – serving as a forum for wider member engagement on Coalition-inspired initiatives or those carried out by individual partners Providing funding – through the Innovation Fund Facilitating implementation – hosting discussions, sitting on advisory boards Encouraging technical rigor – reviewing progress, ensuring appropriate stakeholders are engaged, providing institutional memory on initiatives Advocating for the use of Coalition-associated tools – through the website, annual meetings, strategic communications encouraging advocates in the Coalition to highlight the work of the RHSC A summary of Coalition contributions to key initiatives is at Annex 10. 18 Evaluation of the Reproductive Health Supplies Coalition The main comparative advantage reported to the team was the Coalition’s neutrality: space to meet for jointly and transparently addressing common concerns. Many of the members have had a long history of ‘issues’ with each other and some have been strongly territorial; in the early days before 2009 some used the Coalition as a space to revisit these. Many of the members were also in natural competition with each other, e.g. by being potential recipients of donor funds, or in search of outcomes they could claim. However, before and since 2009 successive Chairs and the Director have pro-actively and constructively worked to alleviate this and build more trust. Even though some of those issues and tensions still exist, suppliers are now able to engage strategically with procurers for example. Trust has also created a more constructive relationship with UNFPA regarding its Global Programme on RH Commodity Security. Operations of the Coalition, leading to some added value for members (as discussed in section 4.2), also have overcome or at least neutralized competition among members. The location of the Secretariat outside the US or the UK has also contributed to a sense of neutrality. This neutrality has been essential to promoting joint accountability, emphasizing the collective interest in making initiatives work and commodity security better. Neutrality is a great asset to the Coalition and its preservation is important. The Coalition has an unusually wide range of constituents – many of the key global players in RH commodity security are represented and some of the key actors at country level. The presence of the manufactures is particularly significant: prior to this they were not engaged with the wider RH supplies community as a group or with some individual members other than through commercial relationships. Their inclusion was strategic and has facilitated considerable gains in commodity security. For example, the Implanon Initiative and the other price improvements is based on better understanding of what manufacturers need to enable more cost-effective production. Other examples of the benefits of membership breadth include stakeholder engagement around pre-qualification and the range of signatories to the HandtoHand Campaign. The Coalition is also notable in its depth of membership from small country-level NGOs to national governments to international donors. Clearly there are issues with how active the government and NGO members are and can be in the Coalition’s present formation. Nevertheless, the opportunity for different constituents to access one another at annual 4. Over and above partner organizations’ own accomplishments, what is RHSC’s comparative advantage, added-value, and value-for-money? 19 Evaluation of the Reproductive Health Supplies Coalition meetings is regarded as a benefit certainly by some of the small NGOs (in particular during the annual meetings in Kampala and Addis Ababa). There is potential for making more of this benefit. This breadth and depth of membership and the Coalition’s neutrality gives it a large and powerful network on which it can draw. According to one interviewee this is slowly being translated into “a community beginning to speak the same language”. This gives the Coalition and therefore its cause, greater weight in the international development arena. Although it has been used on occasion (e.g. to ensure contraceptives were in the UN Commission on Life Saving Commodities for Women and Children), this may be an under exploited advantage. For example, reaching common positions, as was done on the use of pre- qualified commodities, may also be applied for positioning the Coalition as a greater force in international development. The evaluation team recommends that the Coalition consider how it can gain greater leverage from its power as a convenor and use it for advocacy. Convening power also translates into a valuable resource for many members. The annual meetings, though time intensive to organize and attend, are valued by many participants because of the access they gain to other members from different constituencies. As one interviewee said “You’ve got all the players in reproductive health in one place for two or three days – that’s enormously powerful”. Several respondents to the member survey similarly remarked on the value of connections made at the annual as well as regional meetings. The frequent exchange of information through the newsletter, special announcements, and the website are also generally well regarded. The role of the Secretariat in the middle of this gives the RHSC staff, especially the Director, enormous scope, on which they capitalize, to bring people into discussions, raise issues, and convene meetings and discussions in various forms. This is also enabled by the Coalition structure which is fairly flexible and allowed for the opportunistic inclusion of LAC, the development of the cross-cutting Caucus on New and Underutilized Methods and the new Task Force on Maternal Health Supplies. There are issues with how well these modalities work to include southern partners, as discussed later, but at least for the northern partners they work well. The commitment of members was notable among those we interviewed and is evinced by the time that some organizations spend in Coalition activities without immediate or direct return. One senior figure at an international NGO anticipates spending between half to one day a week on his new role as a Working Group head (and believes it would merit two days if his available time was not a restriction). The International NGOs and the technical partners are particularly aware of the costs and benefits to them of membership and clearly make informed and considered decisions about the value of engagement. It will be important to the future of the Coalition that time commitments do not become too onerous. The investment by the bi-lateral agencies is also notable. USAID, for example, had the team interview seven staff who either were or had been very engaged with the work of the Coalition. Even DFID, which is structured very differently with a much smaller technical staff, has two actively engaged staff members, one of whom is Chair of the Executive Committee. 20 Evaluation of the Reproductive Health Supplies Coalition Feedback on the Secretariat and its staff was almost entirely positive. The team evidently works extremely hard and this is recognized and appreciated by members. In most cases they appear to have had the right touch in terms of adopting a largely neutral but constructive position, facilitating the Working Groups, dealing with controversial issues, and managing the politics which were especially difficult in the early days. There is a lack of unanimity over the extent to which the Director should adopt a leadership function rather than just a convening role, but in the majority of instances he and his team seem to hit the right balance. The Secretariat’s limited funding, too small to fully serve current needs, has meant that activities have to be prioritized carefully. These issues are further discussed in section 7. PATH has also managed to maintain an appropriate relationship with the Secretariat and the wider Coalition. Competitors, alert to potential commercial disadvantage, do not seem overly antagonistic towards PATH and do not seem to bear ill will towards the Secretariat because of it. This has helped with the perceived neutrality of the Coalition. The Coalition was funded on principles of volunteerism. While this has been a constraint to participation by some members (further discussed in section 7), it has also reduced the level of competition for access to resources because there have been very few Coalition controlled resources available (essentially only the Innovation Fund). It has also encouraged partners to come forward with funding and projects that can support the work of the Coalition, and has helped keep the Coalition as a neutral space. The vast majority of our interviewees had positive things to say about their experiences with the Coalition. Noticeable were criticisms around access to Innovation Fund grants if they failed to get one. The Coalition has been aware of this and is using the remaining Innovation Funds to support activities by country level organizations. It is wary of the perils of having substantial funds available in a competitive field. Further funding for specific things to be controlled by the Secretariat on behalf of the Coalition may be valuable for the future, as discussed later. However, this should be very carefully managed so that it does not upset neutrality, and thereby increase competition within the Coalition. The Strategic Plan had envisioned that participation in the Coalition would add value in eight different ways. Table 1 gives the evaluators’ assessment of the realization of this added value. 21 Evaluation of the Reproductive Health Supplies Coalition Table 1 Expected added value of participation in the Coalition Expected added value Assessment Providing a forum for technical collaboration and problem-solving. Demonstrated from the outset. Increasingly being used as such by a wider variety of members. Bringing together a “brain trust” of expertise, experience, and learning relevant to finding solutions to RH supply security. Partially demonstrated. Wide and growing membership from different constituents, with effective communications between many of them. However country level members not sufficiently engaged as a resource. Promoting the efficient and effective use of existing, limited resources by improved coordination and harmonization of RH supply programs. Some progress e.g. through CARhs and RHSC engagement in a variety of initiatives, and coordination between partners improved. But greater harmonization not yet evident. Bringing about change through unity of purpose and action. Less emphasis on this in past, but growing. There may be potential for capitalizing on this more. Increasing global awareness of RH supply issues and strengthening advocacy and influence globally and in countries through shared objectives and messages. Demonstrated at global level but less evident at country level. Promoting innovation and collaboration among the public, private, and commercial sectors to develop markets for RH supplies. Not yet demonstrated, but new relationships have been formed which could lead to this. Initial work has been done on total market approaches. Members joining together to seek funds for innovative ideas, research, advocacy, and best practices. Some Innovation Fund awards demonstrate collaborative effort, but there is potential for more. Several RMAWG members are joining together to secure funds to continue advocacy at country and other levels. Maximizing the impact of investments in RH supply security by leveraging the investments of other participating organizations. Demonstrated. e.g. increased funding for RH Supplies from GFATM. Good leveraging from members. This overall added value, and the comparative advantage of the Coalition as a whole, plays out differently for individuals, member organizations, and their combined efforts to address RH commodity security at country, regional and global levels. Added value for individual members and for country level initiatives primarily resides in the access provided to other members and information. According to the members who have worked at regional level, a further major added value to the Coalition has been in the support it has given to overcoming language barriers. (These are further detailed in Annex 11, Added Value of the Supplies Coalition. Comments specifically by southern members are page 24 of Annex 5, Coalition Member Survey 2012 Results Report.) The Coalition has added value to the work of individual partners working at global level by: Changing the landscape in which they work, from wary and sometimes openly hostile to one in which they can work more effectively together. This has provided a space where competitors can collaborate. Bringing down the price of commodities. In addition to the general benefits of this to donors, recipients and manufacturers (because it came with increased orders), it has helped at least one of the bilateral donors deliver on its commitment to reduce the price of RH supplies Contributing to a harmonized approach for quality assurance of health products and how to move forward. Ensuring common ownership of initiatives which might not have been as well received had they been perceived as being led by one donor or technical agency. The reach of tools and initiatives is extended when they are owned by the Coalition. Providing easy access to stakeholders who can be consulted about the development of a tool and support its dissemination. RHInterchange was reported to have developed in this way. The Coalition has also provided continuity in the difficult process of developing new 22 Evaluation of the Reproductive Health Supplies Coalition tools. As one interviewee said “The transition from spawning an idea to implementing it can be a fraught one in terms of how you keep it true to its original impact intent and make it practical organisationally for it to run and operate. The Coalition’s value added is providing that continuity point, so that there can be check in, corrections made, or changes to expectations. That’s really helpful.” This was beneficial for the development of AccessRH. Being a source of intellectual capital on RH commodity security, bringing all the key players and ideas to one place. Acting as a central connecting point between different networks. Providing technical agencies which are usually supported by only one donor, with alliances that reach beyond their traditional ‘camp’. A possible lesson learned is for the RHSC to stay engaged with some initiatives even after they have spun off to other organizations. The neutrality and cohesion of the Coalition could perhaps help to maintain momentum for such initiatives as AccessRH and PGH. In addition, ongoing activities, such as country-level advocacy under the RMAWG, may be linked with longstanding efforts, such as CARhs work to alleviate stock-outs. The table below provides a comprehensive compilation of funding for and related to the RHSC. It does not include for example, the large amount of staff time that is provided by partners to enable the Working Groups to function, or attendance at annual meetings. This cannot easily be quantified and it was outside the scope of our evaluation to try to do so. Table 2 Expenditure on Coalition related activities USD 2009 2010 2011 Donors support to Secretariat 1,304,000 1,897,000 2,086,000 Innovation fund (awards) 712,404 289,419 631,465 Funding for RHSC-related initiatives, e.g. CARhs 7,923,000 10,665,000 23,523,243 Total expenditure on RHSC related activities 9,941,393 12,852,609 26,241,539 It is very difficult to assess whether this overall funding is value for money. The value for money of RHSC-related initiatives spun off from activities started within the Coalition has not been calculated and it is beyond the scope of this evaluation to attempt to do so. It would however be worth considering when detailed assessments of initiatives such as CARhs and AccessRH are carried out. In the absence of this information, there are other ways to look at the issue: Between 2009 and the end of 2012 the four donors to the RHSC Secretariat will have contributed $7.3m. Table 3 shows RHSC Secretariat expenditure for the period 2009 – 2011 with an estimate for 2012. Expenditure shown covers both the dedicated Secretariat team and supporting activities by PATH. For example the line for personnel costs covers Secretariat staffing in the Brussels office, Washington DC, and staff costs charged by PATH for regular and occasional support to the Secretariat. e.g. preparation of donor reports. The other large line item, operations, covers telephone, printing, office costs, insurance, facilities, etc. All other lines are fairly minimal. 23 Evaluation of the Reproductive Health Supplies Coalition Table 3 RHSC Secretariat Income and expenditure 2009 – 2012 USD Yr 2009 (actual)` Yr 2010 (actual) Yr 2011** (actual) Yr 2012*** (planned) Total 2009 - 12 % Expenditure Personnel 966,000 1,185,000 1,412,000 1,455,000 5,018,000 68% Consultants 7,500 106,000 115,000 20,000 248,500 3% Secretariat Travel 82,000 155,000 154,000 115,000 506,000 7% Subcontracts 0 129,000 61,000 0 190,000 3% Annual Meetings 45,000 41,000 24,000 65,000 175,000 2% Participant Travel 16,500 49,000 34,000 55,000 154,500 2% Operations 187,000 232,000 286,000 338,000 1,043,000 14% Total Expenditure 1,304,000 1,897,000 2,086,000 2,048,000 7,335,000 100% Notes on the table: *Plus interest **Additional funds received/spent on annual meeting from BMGF ***Plus approximately $150k in other funds towards forums and working groups Numbers have been rounded and do not exactly match donor financial statements. The phrase frequently used by key informants about the Secretariat was ‘lean and mean’. It has a small staff whose efficiency and effectiveness was praised by a variety of stakeholders. It is generally recognised that they are working beyond capacity. The proportion spent on staff (68%) would seem to be in line with the nature of the Secretariat’s work – largely networking, convening and communicating. The arrangement with PATH enables some specialist functions (e.g. budget administration) to be bought in regularly or occasionally on a fractional basis. Other specialist services such as HR are also available, covered by the PATH overhead. Outsourcing is used appropriately e.g. for website maintenance. Consultants are used selectively for largely strategic pieces of work and this is appropriate. Input from PATH enables the Secretariat to make best use of the donor funds at its disposal. The four donors have different rules about overhead, procurement, staff costs, etc. and the funds have flowed at varying intervals over different periods. The project administrator is skilled in matching the needs of the Secretariat with the funds available , and relieving Secretariat staff of this detailed process. In addition to donor funding, the Secretariat has sometimes mobilized contributions from partners to support ad hoc activities. These are usually in the form of support for travel to events, meetings. etc. Often these are provided directly to simplify processing. The Secretariat is essential to the running of the Coalition. It is the ‘glue’ that keeps it all together and enables partners to be effective in the Coalition ‘arena’. The quality of its work is reported by key informants to be good. Its value for money would also seem to be fairly high. The grants through the Innovation Fund have been fairly small (the largest was less than 200k). (A summary assessment of completed activities with results at country level is at Annex 9.) On the whole those activities seem to have been worth doing. The applications to the Fund were carefully considered, guidance on selection has been followed, and weak applications were rejected. For those applicants who had the capacity to formulate quality 24 Evaluation of the Reproductive Health Supplies Coalition proposals, the application and selection process seems appropriate for the scale of the grants and what they were trying to achieve. Members participating in the internal review suggested a number of improvements in operating the Innovation Fund or something similar going forward. (This is discussed in Section 7.4.2). Reference is frequently made to the leveraging power of the Coalition, its ability to attract resources in support of both its own work and more broadly the cause of RH commodity security. For example in 2011 the Hewlett Foundation, UNFPA, IPPF, BMGF and the Advance Family Planning project contributed to the Access for All conference, LAC and SECONAF activities, and to grants for civil society members of the RMAWG. In the same year DFID allocated funds to the Concept Foundation’s Quality Reproductive Health Medicines project which is being administered under the auspices of the Coalition, and awarded $16m to the CARhs Group for commodity shortfalls. USAID also contributed $1m to this. The following graph shows stakeholder contributions from 2007 – 2011. Graph 1 Stakeholder contributions, 2007 - 2011 In a sense the Coalition is leveraging its own members. On the other hand none need have spent their money on Coalition related activities, and the donors at least could have spent outside RH commodity security entirely. The core investment by the donors to the Secretariat is effectively having a multiplier effect. The Secretariat glues the Coalition together, enables it to function, and in many cases facilitates, or at some point has had a key role in the efforts that attract the funding. This increases its value for money. The most significant demonstration of value for money by the Coalition is the price reductions it has been able to negotiate for large orders. Communication between members who procure led to sharing of information on prices charged and combined negotiation with suppliers. The PGH has also been used in the case of Implanon to access committed donor funds more quickly. Reduced prices have been negotiated for Implanon (saving $7.3m to date 25 Evaluation of the Reproductive Health Supplies Coalition and potentially $15.7 million if the 2012 order target is met), Jadelle (saving $7m to date and likely to be more) and the female condom (saving $359k). The RHSC has therefore saved at least $14m in commodity costs, and potentially $22m. Cost savings for those who procure also benefit those who manufacture: lower prices lead to more sales, greater volume, increased revenues. 26 Evaluation of the Reproductive Health Supplies Coalition The current Strategy (2007 – 2015) has served the Coalition well but is now somewhat out-of- date. Although the 2011 member consultation on the Strategic Plan concluded that it “is mainly on track and in line with current thinking”5, based on key informant input, the evaluators encourage that it be revised to reflect the expanded membership, current conditions, and future possibilities. The Coalition has been well guided by the three goals (see box 1 at the end of this section). In addition, the guiding principles elaborated in the 2009 Terms of Reference have also largely served the Coalition well, although more has been done to encompass method choice than the status of women and equity. There has been some drift from or lack of progress on the principle of country ownership in part because some RHSC operations (e.g. the functioning of the Working Groups, principle of volunteerism) have not enabled full participation and therefore ownership by countries. In fairness, membership criteria were restrictive early on and only recently has country-level representation increased to become the largest share among members. In contrast, principles relating to mutual accountability and functioning as a forum have largely gained momentum. The evaluation team detected reasonable congruence between the overall direction of Strategic Plan and the functioning of the Coalition over the last five years since the plan was written. (The RHSC Principles are discussed and analyzed further in section 6.1.1 of this Report.) In practice the ‘interactive overview’ matrix version of the Strategic Plan, which was developed by the Director to facilitate communicating it internally and externally, has probably served as a better strategic guide than the Plan itself. Although there was some criticism of having ‘dumbed down’ the strategy, the matrix presentation is a concise and easy way to understand the Strategy, and likely has more relevance to members than the lengthy document itself. However it is now rather out of date and there is a problem with the metrics (not specific to the matrix) as discussed in the following section. In past review documents such as the 2011 internal member survey, and in the course of this evaluation, various strategic issues have been raised. Among these are the role of the Coalition in addressing demand for commodities; the extent of the focus on country-level support to achieve Coalition goals; and whether the RHSC should address all RH supplies or mainly family planning supplies. In practice however these have not yet played out to create misalignment with the Strategic Plan nor have there been major tensions within the Coalition. There is now however a need to revisit the strategic goals, principles, and assumptions to address new challenges and opportunities and to enable some of the above issues and other strategic approaches to be considered for taking the Coalition forward. _________________________ 5 Kinzett, Steve (January 2011). Member Survey on the Strategic Plan, p9 5. To what extent is RHSC’s role adequately reflected in its strategy, theory of change and M&E framework? 27 Evaluation of the Reproductive Health Supplies Coalition A definition of a theory of change is given in Box 2. There is no such explicit theory of change within the RHSC to underpin the strategy. In various points of the Strategic Plan a theory is implied. One can draw links between the ”gaps, challenges and opportunities for meeting RH commodity needs”, the rationale for the Coalition, and the added value that participation in the Coalition is intended to bring. In practice the evaluators found that there was broad agreement among members on what the Coalition should be trying to achieve and how it should be trying to do that, again implying a common theory of change. However this is not articulated in one place. A process of developing a theory that is participatory (as far as is practical) should be used when the Strategic Plan is revisited. This would have the advantage that: The strategy would be clearly based on achievable objectives, which is not the case currently. This is discussed below, section 5.3. Members would ‘buy in’ to these objectives. There would be clearer links between what the Coalition is doing and what is being measured. The absence of an articulated theory contributed to some problems with the MEF that are discussed below. Risks and assumptions would be made more explicit. Cohesion and common understanding among members of the role and potential of the Coalition would be further developed. The Coalition might find it useful to supplement its eventual theory of change with a logical framework. Log frames are useful for creating well-articulated links between overall goals and inputs. However, they can be problematic for example if used as a rigid monitoring tool. As with a theory of change, the process of developing one would help to resolve differences in stakeholders’ understandings about what is being done and why. The Strategic Plan of 2007 contained a list of 58 illustrative indicators to suggest the kinds of information that could be used to monitor progress of the nine objectives under the three goals (see box 1). A monitoring and evaluation framework (MEF) was then developed, refining a list of 30 specific, operationalized and collectable indicators. These indicators had baseline values (mostly set for 2007), a strategy for collection, and a means of verification. It was also agreed that indicators should already be collected by one or more Coalition partners (or agency of a partner) and they should be relevant to 14 focus countries receiving targeted support by the Reproductive Health Interchange (RHI), Project RMA (Resource Mobilization Box 2: Explanation of theory of change “A theory of change (TOC) is a tool for developing solutions to complex social problems. A basic TOC explains how a group of early and intermediate accomplishments sets the stage for producing long- range results. A more complete TOC articulates the assumptions about the process through which change will occur and specifies the ways in which all of the required early and intermediate outcomes related to achieving the desired long-term change will be brought about and documented as they occur.” Adapted by Erin Harris, Research Analyst, Harvard Family Research Project, from Anderson, A. (2005). The community builder's approach to theory of change: A practical guide to theory and development. New York: The Aspen Institute Roundtable on Community Change 28 Evaluation of the Reproductive Health Supplies Coalition and Awareness), and the Global Programme of the United Nations Population Fund (UNFPA). The intention was to review the Strategic Plan and MEF in 2009 at the end of a grant from BMFG, and then again in 2015 to coincide with the end of the Strategic Plan. At the request of the evaluators, in 2012 the Secretariat kindly prepared another monitoring report to inform their judgements (see Annex 13). In addition to these occasional reports there are also annual narrative reports to the four donors compiled by the Secretariat and PATH. BMGF has one format and the DFID, USAID and UNFPA have a very similar one, but both contain effectively the same information. These report against Goal Three which covers the work of the Secretariat. In the report to BMGF the Secretariat reports against five objectives (different from those in the MEF) and 13 activities (see Box 1). These have remained unchanged since 2009, with the exception of a fifth objective and some associated activities in 2011 to accommodate the Hand to Hand Campaign. The reports provide a good summary of the work of the Secretariat and the system seems to work adequately. There are neither adequate systems in place to monitor and report against Goals One and Two nor is the MEF measuring the right things. The scope of the MEF is too wide. The MEF is essentially trying to take the temperature of the state of the world’s commodity security while the Coalition can in practice only affect some aspects of it. As a result there is a mismatch between what the Coalition is doing and what is being monitored. For example Goal One, (Increase availability, predictability, and sustainability of financing for RH Supplies) is being addressed by the Coalition through the PGH, which has created added value out of existing resources, and the Hand to Hand campaign, which aims to generate new resources. However the indicators measure completely different things, such as how many focus countries are fulfilling their spending plans for RH supplies. It should also be noted that the focus countries no longer have great meaning within or relevance to the RHSC. These countries were originally selected on the basis of donor priorities, not Coalition aims. Some of these disconnects arise from lack of clarity about what the Coalition actually IS and what it DOES. On the one hand, the RHSC could be the sum of all its members, including some national governments, in which case an increase in expenditure on contraceptives by those governments could be said to be the work of the Coalition. On the other hand, it could be defined as simply the Secretariat, in which case only initiatives such as the Hand to Hand Campaign should be monitored. Neither position is completely correct. What the Coalition is and does has to lie somewhere in the middle, such that it can faithfully claim credit for work that its partners come together to implement. The interactive overview on the RHSC website comes closer to reflecting the actual work of the Coalition as linked to the Strategic Plan (or at least it did: it is rather out of date now). However it falls down when it comes to ‘how do we measure success?’ because it relies on the MEF indicators. A review of this website presentation makes the mismatch very transparent. 29 Evaluation of the Reproductive Health Supplies Coalition The MEF does not fully capture what the Coalition has achieved. Although the Coalition is undertaking some strategic initiatives which are very relevant for the achievement of the goals, they cannot all be reported against the MEF other than stating that they exist. Performance is captured through reports to the donors but that many key results do not appear in the MEF means that many of the achievements of the RHSC are difficult to establish without extensive document review and consultation. In addition the MEF does not capture regional achievements. The MEF is not used as a management tool. The evaluation team asked if another report against the MEF could be produced to contribute to this evaluation. This the Secretariat and its partners willingly did. However it was apparent that the existence or absence of an up-to- date monitoring report made little difference to the Secretariat, its members, or its donors. The donors’ needs are met by the annual report which details the work of the Secretariat against a separate set of objectives from those in the MEF. These annual reports also recount some of the wider Coalition achievements. Various other reports are provided to the Secretariat team on the progress of some of its initiatives, e.g. the CARhs team sends an annual report to the SSWG for review and discussion, and these prompt a response from Coalition members. However the MEF is neither a summation of all these reports, nor is it used as a way of assessing progress against its strategic goals or objectives, probably because no one finds it useful and it is a huge effort to gather the data. At the same time, the MEF with its broad scope is not used to assess commodity security more broadly, further diminishing its usefulness as a strategic tool. The MEF does not promote accountability. A well designed MEF should make it possible to link from goal to objective to activity onto workplans so that there is accountability for delivering against that MEF. However this is missing from this MEF. There may be value in some form of common monitoring of commodity security overall by the Coalition. An ongoing process of gathering data from multiple partners and tracking across time could be valuable for advocacy, resource mobilisation, and informing Coalition strategy. However, the added value of the Coalition undertaking this sort of exercise, beyond partners’ work in this area, would need to be considered carefully. In addition, this should be clearly separated from the MEF. The MEF needs a wholesale review. The three Goals are representative of what the RHSC is ultimately trying to achieve, but the objectives need to be reframed as outcomes (thus letting them sit as high level aims not to be achieved by the Coalition alone). In addition, new lower level objectives should be formed which have a closer relationship to the work of the RHSC and which imply accountability for their delivery. This would also enable a link to be formed between the new outcomes (the old objectives) under Goal three and the objectives against which donor reports are made and which at the moment are floating free of the MEF. This should also make it easier to translate the MEF into the workplans of the Secretariat and the Working Groups. All this will be made easier by developing a theory of change. Attention should also be given to trying to make data collection much less arduous and this should be considered when selecting indicators. The Executive Committee and the Director are aware of these challenges and the job description for the Monitoring and Evaluation Officer charges that person with addressing these issues. 30 Evaluation of the Reproductive Health Supplies Coalition Box 1: Goals and objectives for the RHSC Strategic Plan goals and objectives Goal 1: Increase availability, predictability, and sustainability of financing for RH Supplies. • Objective 1.A: Increase flow of new resources. • Objective 1.B: Ensure that countries and their governments make use of the resources already at their disposal. • Objective 1.C: Mobilize Total Market Resources Goal 2. Strengthen capacity of health systems to deliver RH supplies in a sustainable manner • Objective 2.A: Ensure the integrity of systems for supply chain management, both globally and at country level. • Objective 2.B: Provide a coordinated, short-term response to stock outs, supply disruptions and other failures that occasionally befall functional supply chains. • Objective 2.C: Promote effective delivery of RH health supplies in settings where routine systems for supply chain management have broken down or ceased functioning. Goal 3: Assure the added value of the Coalition as a productive and sustainable global partnership through efficiency, advocacy and innovation • Objective 3A. Foster greater harmonization among stakeholders at global and country level to generate economies of scale, maximize impacts, and minimize the risks of redundancy and confusion. • Objective 3B. Generate, disseminate, and facilitate the application of knowledge through innovation, research, and documentation. • Objective 3C. Offer a credible and sustainable institutional framework within which stakeholders can collaborate, build commitment to supply security, formulate strategies for change, develop common indicators of success, and measure progress against them. Annual monitoring objectives in the report to BMGF Objective 1: Ensure the functionality, smooth operation, and efficiency of the Coalition. Objective 2: Ensure that the Strategic Plan and associated program and policy frameworks (three-year work plan, M&E Framework, communications strategy) continue to shape and facilitate the Coalition’s work. Objective 3: Maximize the opportunities afforded by an expanded membership to further the Coalition’s effectiveness. Objective 4: Ensure the financial sustainability of the Secretariat and the Coalition. Objective 5: Support the Alliance for Reproductive, Maternal, and Newborn Health in contributing to the Coalition HANDtoHAND Campaign’s goal of reducing by 100 million the number of women with unmet need for contraception. 31 Evaluation of the Reproductive Health Supplies Coalition In this section, we review the structure and operation of the RHSC as it was and what it has become. The 2011 Julie Solo Report ably summarizes emergence of the global movement for reproductive health commodity security. Initiated by a dedicated group of activists and extended to others during the 2001 Istanbul Conference, the purposefully named Interim Group on Reproductive Health Commodity Security sustained this movement over the next several years. Midway through the decade the Interim Group gave way to formation of the RHSC. Interestingly, some of the same core components critical in the early phase remain central today: small groups of activists, ongoing discussions and deliberations among them, and global meetings to extend the movement to others. Building on what came before, the RHSC has evolved. Since 2006, under the overall guidance of an Executive Committee, a small Secretariat has been in place to support the membership and the three Working Groups. By 2009, RHSC had embraced extending membership to all organizations interested and aligned with the RHSC’s vision that “all people in LMI countries are able to access and use affordable, high-quality supplies, including a broad choice of contraceptives, to ensure better reproductive health.” By 2010 the funding base of the RHSC had expanded beyond the Gates Foundation, which had supported RHSC through its formative years, to a coalition of funders now including DFID, UNFPA, and USAID. The 2009 Terms of Reference (TOR) contains a set of guiding principles and a diagram of the structure within which the RHSC operates. As such, the TOR provides a relevant and useful roadmap to review how the RHSC has developed and expanded over the years. The RHSC’s guiding principles contained within its 2009 Terms of Reference (TOR) derive in large part from the set of best practices for global partnerships for health. In 2005, the High Level Forum (HLF) on the Health MDGs set out principles for best practice in global health partnerships, Building on the Paris Declaration on Aid Effectiveness and remaining useful today, these best practices are summarized as follows: Ownership: Global Health Partnerships respect partner country leadership and help strengthen their capacity to exercise it Alignment: Global Health Partnerships base their overall support on partner countries’ national development strategies, institutions and procedures Harmonization: Global Health Partnerships’ actions are more harmonized, transparent and collectively effective; Global Health Partnerships collaborate at the global level with other partners to address cross-cutting challenges such as health system strengthening Managing for Results: Global Health Partnerships work with countries to adopt and strengthen national results-based management 6. In what ways and to what effect has the structure and operation of RHSC evolved, particularly since 2009? 32 Evaluation of the Reproductive Health Supplies Coalition Accountability: Global Health Partnerships provide timely, clear and comprehensive information6 The 2009 TOR lists seven guiding principles as critical for “achieving a sustained supply of affordable, quality reproductive health supplies in low and middle income countries”. Summarized from the TOR, how each has since been applied is reviewed below: “The RHSC understands that achieving (its) vision requires (optimally applying) public and private resources…and utilizing a multi-stakeholder approach to (serve people’s RH supply needs and) improve RH behaviors” – The RHSC has become the largest extant body dedicated to FP/RH. Opening membership after 2009 to all organizations aligned with its mission “to ensure that every person is able to obtain and use RH supplies” has made this possible. Currently approaching 200, RHSC member organizations cover a broad range: small country-level civil society organizations and large commercial manufacturers; private foundations; government donor agencies; and multi-lateral donors; developing country governments and social marketing enterprises; supply system expert groups and generic producers of particular products. “The RHSC acknowledges that (effectiveness results through) complement(ing) the actions of its individual members (through leveraged, collective action; promoting collaboration and cooperation); and work(ing) by consensus towards common goals” – Structurally, the RHSC is well positioned to promote collaboration and combined action among its members. Practically, however, much remains to be achieved. The membership has grown beyond and not been sufficiently integrated within the original structures. As a result, many opportunities for collective action towards common goals are missed. “The RHSC believes that increased country ownership is fundamental…(and) will work through national governments to develop supportive policies, plans, resource commitments, and capacities” – Country ownership for the RHSC remains a principle not a practice. While membership at country levels has increased, regular involvement within RHSC structures has not. Furthermore, the few governments approached – as part of holding RHSC annual meetings in Uganda and Ethiopia – remain exceptional, not continual. “The RHSC recognizes the role of both industrialized and developing-country manufacturers, for planning, providing, and delivering quality RH supplies” – The RHSC has not only recognized but now also encompasses manufacturers within the Coalition. Three major pharmaceutical companies have joined the RHSC as well as a number of generic manufacturers. This opens the way toward private-public sector dialogue, edifying for both sides. “The RHSC strives to ensure greater equity and improve access to RH supplies for those who are poor and marginalized” – This remains the North Star for many Coalition members. Some global improvements, such as stock-out alerts and better data on procurements, eventually reach such populations. However, efforts to more directly impact those in need have been few within RHSC. Especially within the broadened perspectives of private, commercial sector engagement within the RHSC, this is an important principle to retain. “The RHSC recognizes that (diversity of) needs (requires making) a range of products (available within a client-centered approach, including bringing) attention to underused RH options and the research needed (to meet) contraceptive method gaps” – Member concerns with these issues have surfaced and received some focused attention, particularly within the Caucus on Under-used Methods, the Task Force for Maternal Health _________________________ 6 Best Practice Principles for Global Health Partnership Activities at Country Level, High Level Forum on Health MDGs, 14-15 November, 2005, Paris 33 Evaluation of the Reproductive Health Supplies Coalition Supplies, and the MDA Working Group’s recent outreach to reproductive health product developers. “The RHSC recognizes (that disadvantages in) women’s status, resources, and decision- making (undermine RH commodity security)” – This remains central within a movement originated, as Julie Solo reminds us, “(by) a small group of committed and driven women”. Especially within a structure that lends itself more to technical discussions than practical solutions is retaining this principle important. RHSC is an operational entity that is subsumed under PATH, which receives and manages funds designated for the RHSC. The decision to not officially register RHSC is practical as well as purposeful. Moving the funding through PATH was useful for the Gates Foundation, its founding supporter, which already had an established funding mechanism with PATH. As other funders joined to provide collaborative funding for RHSC, they have continued to channel funds through PATH. With PATH providing backup management and administrative support, particularly through the start-up years, RHSC was able to quickly take form from 2006 onwards. With guidance from the Executive Committee, staff members were soon recruited and the Secretariat established in Brussels. The structure within which the RHSC works to support Coalition members and the work on commodity security is detailed in the following. The Executive Committee is the governing board of the RHSC, a committee rather than a board because the RHSC is not an officially registered organization. In scheduled bi-annual meetings, the Executive Committee reviews past accomplishments, assesses the present situation, and sets future directions for the coming year. The Executive Committee includes all four funders of the RHSC, heads of the three Working Groups, and representatives of organizations with notable presence in international commodity security and family planning, namely the International Planned Parenthood Federation (IPPF), the Pan American and West African Health Organizations (PAHO and WAHO), and the World Bank. Executive Committee members, including the funders themselves, noted that the donor representatives form an inner core and exercise correspondingly greater influence within the Executive Committee. (An exception is the World Bank, which has not been notably active in the RHSC since serving as the Executive Committee’s first Chair.) Chairpersons in particular, historically also from the donor community, have been particularly influential in shaping the RHSC. Though this risks becoming a donor-driven entity, to date this has been ably counterbalanced by recognizing that the RHSC is a coalition of its members. Key informants at various levels encouraged expanded representation on the Executive Committee. Among specific recommendations offered were to include new categories representing member constituencies, such as manufactures and country-level organizations. In addition, one well-placed key informant suggested deepening discussions and deliberations within the Executive Committee by adding independent experts on commodity security and international family planning. The Secretariat is the operational center of the RHSC. The RHSC’s group of core staff has turned over some since 2006, but remains small and focused on supporting the Coalition 34 Evaluation of the Reproductive Health Supplies Coalition members. Secretariat staff members give special attention to the established structure of the three Working Groups. In addition, Secretariat staff support the Caucus and the Task Force, which have emerged as venues to represent and express member interest in particular areas. That the Secretariat has operated under the same Director since 2006 has provided consistency even as the RHSC has grown. The three Working Groups are designated structures of the RHSC through which Coalition members engage particular substantive issues: Systems Strengthening; Market Development Approaches; and Resource Mobilization and Awareness. Enshrined in RHSC founding documents, all three Working Groups built on work already initiated under other projects. For example, Deliver as USAID’s lead organization for strengthening supply chains has been involved in efforts to improve FP commodity data collection and dissemination well before the RHSC came into being, initiating early versions of the RHInterchange as well as the CARhs. Similarly, the Interim Committee on Reproductive Health Commodity Security , funded by the Gates Foundation from 2003 to 2006, was a precursor not only to the RHSC generally but also to the Resource Mobilization and Awareness (RMA) Working Group. Earlier interest in market segmentation for tiered pricing of contraceptive commodities provided precedent and momentum for the more broadly conceived Market Development Approaches Working Group, which focuses on equity as well as access. During the early, initial years of the RHSC, member organizations were well represented within each of the three Working Groups. However, with recent increases in RHSC members, the Working Groups have not correspondingly expanded their reach. The Working Groups thus risk becoming more an elite sub-culture of founding members than a vital and vibrant part of expanding interests and possibilities. If only by historical precedent, the primacy of the three established Working Groups seems largely unchallenged. An evolving RHSC, embracing its broader membership and making strategic adjustments, may need to reconsider the current structure and adapt to serve evolving needs. Beyond the Working Groups, members have found need for and formed other structures to pursue particular interests. This clear evidence that the Coalition is member-led also demonstrates the Secretariat’s role and willingness “to lead from behind”. However, like the Working Groups, such structures as the Caucus for New and Underutilized Methods and the Task Force for Maternal Health Supplies have not yet been an avenue to expand country- level participation. The Regional Forums in contrast have potential for wide as well as deep involvement at country level. However, the list of regional forums which never made substantial contributions at country or other levels is long. These various RHSC structures, all of which receive support from designated Secretariat staff, are briefly described below: Interest among a few members in giving special attention to certain contraceptives and other RH products led to formation of the Caucus on New and Underutilized Methods. Discussions beginning as early as 2006 eventually led to forming a group under the umbrella of the RHSC. Different from Working Groups, the Caucus did not operate within work streams or develop an annual work plan. Instead, the Caucus meets less frequently, has regular conference calls, and is intentional about trying to influence the three Working Groups. The Secretariat saw value in having this added focus and organized to provide backup support for Caucus discussions and its annual face-to-face meetings. 35 Evaluation of the Reproductive Health Supplies Coalition Expanding RHSC beyond contraceptive commodities similarly grew out of member interest, including discussions that occurred initially within the Caucus. Focus on maternal health gained impetus during the 2011 Annual Meeting in Addis. Encouraged by a participating representative of a private foundation with longtime strategic investments in maternal health, a special lunch roundtable was convened to discuss maternal health supplies. Building on this expression of member interest, the Task Force soon emerged. Deciding to focus upon three critical commodities – magnesium sulfate; oxcytocin; and misoprostal – gave the Task Force a unique position among other groups outside the RHSC similarly working on maternal health. As with the Caucus, the RHSC quickly recognized the value in this and assigned a Secretariat staff member to support Task Force operations. Interest in bringing the LAC Forum into the RHSC emerged out of the regional meeting in Panama in 2010. As one key informant who was there observed, “great diplomatic skills” of the RHSC Director helped to make this happen. As a donor representative noted, occurring at a time when donors were withdrawing from the LAC Region, the Forum is a useful way for countries in that Region to stay connected with the field of commodity security. Similar to the RHSC itself, international organizations appear to be the drivers within the LAC Forum. UNFPA and USAID have made some funding available for the LAC Forum to operate its own Innovation Fund in support of regional Working Groups. The most recent RHSC newsletter listed the first round of awards made. Similar to what transpired in the 2010 Panama Meeting of LAC, a recent regional meeting in Burkina Faso was pivotal for bringing the Francophone Africa regional forum, SECONAF, under the umbrella of the RHSC. Again, central leadership of the RHSC had a key role in bringing this about. The subsequent decision by the Hewlett Foundation to consider making a grant through the RHSC for SECONAF should create some momentum to build upon. RHSC’s courting of the French government, which will host the next annual RHSC annual meeting, could also prove strategic and useful. The Secretariat experienced some staff turnover and consequent disruption during the last year. Heavy staff workloads, including tremendous effort involved in backstopping RHSC activities and events, such as the 2011 Annual Meeting in Addis, may have taken some toll. The Secretariat has come through these changes, but in the process lost some technical expertise. A number of key informants remarked that greater technical expertise would usefully add to the current configuration of the Secretariat. Two new staff positions are being filled within the Secretariat. One is already in place and handling communications within the RHSC, which had until recently been outsourced to a part time consultant and supported in-house by PATH. The second position will be for Monitoring and Evaluation, recognizing that RHSC has not given this consistent attention over time. 36 Evaluation of the Reproductive Health Supplies Coalition Though not explicitly stated as operational policy, the Secretariat has encouraged leadership turnover within each of the three Working Groups. In 2012, a convening by the Secretariat of Working Group heads encouraged that each Group undertake a visioning exercise to confirm or refocus direction for the Group. Sensing that Working Groups were not as vitally involved in the mission of the RHSC as they once were, the Secretariat has sought to reinvigorate them. Particularly during these Working Group transitions, back-up support provided by designated Secretariat staff is crucial. As expressed by many key informants, this support from the Secretariat is also greatly appreciated. Though the relevance of work streams received attention in subsequent SSWG or the MDAWG meetings, focused attention to conduct a visioning exercise did not occur within either. Consultation with and assistance from professional meeting facilitators would usefully contribute to such RHSC convenings, including annual meetings. More broadly, and particularly within the context of updating its Strategic Plan, the RHSC needs to consider how these established structures will serve the new and emerging opportunities provided by a vastly expanded membership. PATH continues to provide back-office support to the operations of the Secretariat. This remains primarily within the areas of financial and human resources management. PATH has also been able to assist in completing particular RHSC tasks when requested. As both a member of and fiscal agent for the RHSC, PATH needs to continue to avoid conflict of interest areas. Keeping the RHSC as several termed it “lean and mean” appears highly prized. Many key informants were pleased and surprised that the Secretariat is able to do so much so well with so few. Furthermore, key informants, and particularly donors, expressed interest in keeping the RHSC trim, even while several saw need to add some technical depth to the Secretariat . Some key informants, including within the Secretariat itself, recommended adding a deputy director. Other key informants suggested that expertise on policy, advocacy, and country-level programs would be useful additions. Many key informants attested to the usefulness of the Innovation Fund and some suggested making this or similar funds, as is now occurring on a limited scale with remaining Innovation Funds, also available for selected country member organizations. Current operations and structures of the RHSC are further assessed in the next two sections, Section 7 and 8. 37 Evaluation of the Reproductive Health Supplies Coalition The evaluators visited Brussels, spending the better part of four working days within the RHSC office. In the process, they logged more than fifteen hours interviewing Secretariat staff, including over ten hours with the Director. A follow-up interview with the Director took place in London and the evaluators also interviewed the half-time RHSC staff member based in PATH/Washington DC. Three staff members of PATH/Seattle, the fiscal agent, RHSC’s administrative backstop, and a Working Group member, were also interviewed. In addition to materials available on the RHSC website, the evaluators also requested and received from both Brussels and Seattle a number of documents relevant to RHSC structure and operations. This information, together with member responses to questions about the efficiency and effectiveness of RHSC operations, provide the basis for assessing what has worked well and what should be improved within RHSC. As Director, John Skibiak received a great deal of praise from Coalition members interviewed. Almost all key informants offered high compliments about his work and what he has been able to accomplish. One key informant from a large international NGO said: “The director, John Skibiak is a strong person…since he’s been on the job he has learned a lot and increased his network and he has done what from my distance seems a very good job”. Another offered this perspective: “John Skibiak…is a very good person to (head the RHSC) because he’s broad and can work with a large set of people in a way that is helpful for bringing in different perspectives”. And from another: “John that's been the driving force behind (raising the prominence of FP/RH commodity security)…he's got involved in a whole range of forums which we never used to get involved in. But the problem is he has a limitation – he’s one man!” One key informant more expansively observed the following about the RHSC Director’s evolving leadership: “In the last 24 months, the Coalition…as an entity, in other words John himself, has…felt confident enough and has had the opportunity to get out and about as the Executive Director of the Coalition, with the feeling of strength and begin to engage at the international level, you know, at the table as the Executive Director of the Coalition. I think it is a relatively new phenomenon, but a result of the support that he has been given by the key stakeholders of the Coalition, who feel that supporting the Coalition and working through the Coalition is one of the key ways in which they will meet their own internal strategies for reproductive health.” Assessing the Director’s leadership and what he has done with the resources available to take the Coalition in interesting new directions, the evaluators conclude that he has been just the right person to lead the RHSC through this period of rapid membership growth. If any criticism can be levied, and it is one against the Executive Committee as well, it is that too much has been taken on without sufficient time and attention to ensure alignment with the 7. What works well and what needs to be improved or changed about RHSC’s internal structure and external operations? 38 Evaluation of the Reproductive Health Supplies Coalition RHSC strategy and members. As a private foundation staff member, herself a long-time participant, noted: “On the (RHSC) strategy, we do have a strategy but… 2007 is when it was developed and it's more aspirational … it doesn’t reflect the current need, and this push and pull between how (the Secretariat) can just, you know, help to orchestrate all their members and let them lead, or (that it) need(s) to take leadership in (some) areas.” Especially during the last few years has the RHSC been moving quickly forward, and it has done so with an outdated strategy that reflects neither changes that have occurred nor new opportunities that have opened. Keeping followers and future directions aligned is a leader’s ultimate responsibility. In contrast, as reviewed in the previous section (Section 6) on how the RHSC has evolved, the guiding principles set forth in the 2009 Terms of Reference are less time-bound and still well able to represent how the Coalition aspires to operate. Key informants perceived Secretariat staff members, including the Director, as dedicated, supportive, and hard working. Many respondents expressed great appreciation for how much so few are able to do. Several expressed surprise that the core staff was as small as it is: four based at the Secretariat in Brussels and one half-time staff member in Washington DC. (Within the last month, the Brussels-based Secretariat added a communications staff member; an M&E specialist is also in the process of joining the Secretariat.) Without exception, key informants in direct contact with Secretariat staff were complimentary about help received and relationships developed. Organized to assist the Coalition Working Groups and other structures (i.e. the Caucus, Task Force and the two regional forums), this small staff is well organized, highly motivated, and over-worked. Some evolution in the amount and level of support provided is occurring, from being responsive to what a particular group may need or request, to being more proactive to ensure that a particular group moves along. As such, Secretariat staff members assigned to Working Groups have begun to function as co-leaders when needed, though clearly deferring to the designated and elected group leader. The various groups, and in particular the RHSC’s formal structure of Working Groups, have over the years experienced ebbs and flows. Based on voluntary participation by members, all of whom have other fulltime responsibilities within their own organizations, Working Group members and leaders alike report being stagnated at times and more engaged at others. In contrast, both regional forums are relatively new and have not as yet experienced such ups and downs. The Task Force, which came out of discussions among Caucus members and during the 2011 annual meeting in Addis, is also new. The Caucus, which serves a cross- group function as well as leadership on a particular issue, seems to maintain a sustained level of interest in new and underused methods. Overall, the willingness of members to sustain such groups is a measure of the value they gain from participation. Several private foundations, among them the Gates Foundation, the Packard Foundation, and the Wallace Global Fund, supported the movement to improve contraceptive commodity security during its formative years, 2000-2005. The Gates Foundation continued its support enabling the RHSC to emerge to sustain this movement. In 2009, three other donors joined 39 Evaluation of the Reproductive Health Supplies Coalition with Gates to continue funding for RHSC through 2012. (In addition, the Gates Foundation has notably stepped in when needed with supplementary support to establish the Innovation Fund and to expand the Addis Meeting.) The four donors have been purposively penurious in supporting the RHSC. A small Secretariat, “lean and mean” as several stated, seemed to make sense for supporting a “coalition of the willing”. Furthermore, the RHSC was not envisioned as a vehicle for making funds available. As the Director stated, when asked if the RHSC should have substantially larger amounts of funding to spread among partner organizations, “What you wouldn’t want is a pot of money so that organizations now feel they have a feeding trough to fund their own activities that they would have done anyway.” Relatively small amounts provided through the Innovation Fund apart, the RHSC has not aspired to be a Global Fund or Global Programme. Rather it works through what member organizations are able to contribute by virtue of mutual interest in particular subject areas and what members may be able to attract in kind from within their own organizations and/or in funding from other donors. This is what has worked in the past. Whether this same model is the one to carry into the future is discussed under the Questions addressed in Sections 9 and 10. As indicated above, the dedicated staff of the Secretariat is stretched thin. The lean could well become mean as the pace continues and burnout occurs. Staff recently added for communications and M&E will not reduce significantly the work burdens of current staff, since related work that was previously out-sourced (largely to PATH) will now be handled within the Secretariat itself. An important addendum to the operations and structure of the RHSC is that volunteerism has limits. Considerable burden falls in particular on those who lead the Working Groups. In addition, effective action along designated work streams of Working Groups often requires considerable effort beyond an individual’s normal scope. As a result, exciting ideas, projects, and studies generated within Working Groups sometimes, if not often, fall by the wayside. The Innovation Fund has, in this situation, proven a useful mechanism by making some additional resources available for particular activities. Another constraint is that the large numbers and especially many newer members have not meaningfully contributed through the Working Groups and will likely not be able to do so in the future. Even without the added complications of geographical and financial distances, the existing structure of three Working Groups could not absorb or deal with vastly expanded participation. Indeed, most Working Group respondents indicated that they have between 15 and 30 members who participate somewhat regularly, with an inner core of 4-6 who are highly involved and dedicated in discussing and moving particular subject areas and agendas forward. In addition, there are other considerations with volunteerism. What participation contributes to individuals is as important as what members contribute. Respondents to the member survey undertaken as part of this evaluation provide some insights on involvement in the Coalition. The total number of responses to the electronic survey of 12 questions was 65, a self- 40 Evaluation of the Reproductive Health Supplies Coalition selected one-third of all the RHSC organizations to which the survey was sent. This is itself a reasonably high response rate for a survey of this kind, especially considering that more respondents reported only moderate levels of active participation (at least once a quarter) than any other category. Furthermore, more than one-third of respondents reported that they had no affiliation with any RHSC group. Though the majority of respondents work within INGOs, fully one-third reported that they were part of country-level organizations. Despite being far removed from active participation, many seem to gain from being a part of a broader movement. (A Summary Report of the Member Survey is in Annex 5.) However, it may well be a missed opportunity that current RHSC structures do not actively engage more member interest, particularly those who are at country level. Similarly, the large international organizations, many of which have been with the RHSC since its founding, do not themselves sufficiently involve staff working at ground level, except for sponsoring some to attend an annual meeting. As a result, the Working Groups continue to function as they always have - at a high, largely headquarters level, focused on global concerns and advances. Encouragingly, the recent MDA Working Group meeting made a concerted and largely successful effort to reach beyond its usual participants. Practical implementation at country level does occasionally occur as a Working Group activity, usually with Innovation Fund support. However, this is driven more by a global perspective on what is needed to advance the field than practical consideration of what would benefit a particular country. Finding opportunity for country-level input to move commodity security forward would usefully add to what the RHSC Working Groups are able to contribute. In addition and alternatively, new structures focused on particular countries or groups of countries (the latter to facilitate cross-country interchange and learning) may be another operational approach to consider. It may also be that the model of the “Working Groups as the engines of the Coalition” is itself outdated and outmoded. With leadership being sometimes exercised from the top and the convener or facilitator role not always clear, perhaps an entirely different structure for the RHSC is needed to continue operations. This is a matter better left to the members in subsequent discussions as to who they are now, where they want to go together, and how they may get there. In other words, when they next update and revise their Strategic Plan and when they logically link RHSC and member inputs with outcomes expected. The RHSC, most notably through the Director and especially over the last two years, sometimes goes beyond the existing structure to investigate new areas of potential engagement. As occurred with contacts and discussions with several manufacturers, this may result in new members and perspectives within the RHSC. Such prospecting was also instrumental to bringing the two regional forums under the umbrella of the RHSC. On other occasions, such as a recent meeting in Bellagio on long-acting reversible contraceptives for developing countries, co-sponsored with the Population Council and the Federation of International Gynecologists and Obstetricians, the RHSC increases its visibility and possibly its reach into new areas of substantive inquiry. Remaining informed about perspectives and understandings not currently receiving attention within the RHSC will help guide future operations. It is often difficult to separate out the contributions of member organizations from those of the RHSC. Clearly the RHSC enters a gray area when it claims some achievements of members 41 Evaluation of the Reproductive Health Supplies Coalition as at least partly its own. Members themselves were sometimes divided as to which organization should get credit for what. For example, some key informants thought that RHI and CARhs would have happened independently of the RHSC. Others gave credit to RHSC for extending and publicizing these same initiatives. One key informant, a Working Group leader, suggested that such contributions exist along a continuum ranging from spawning or germinating an idea, to actually developing it (usually done by a single organization), to promoting and spreading its usage, to providing some ongoing maintenance. In the case of the Pledge Guarantee for Health, this respondent viewed RHSC as having made contributions from the earliest stage. In the cases of RHI and CARhs, RHSC was on the other end of the continuum, encouraging their use and ongoing maintenance. Attribution is similarly murky with regard to funding claimed as leveraged by the RHSC. In some cases, key informants who were directly involved, such as for AccessRH and Pre- Qualification thought that RHSC was instrumental for their organizations to attract funds to take initial Innovation Fund studies to scale. Others offered more skeptical assessments akin to a faulty syllogism that organizations routinely count as leveraged everything within reach. In some areas however what was birthed within the RHSC may have wide-ranging impact. For example, AccessRH, especially if combined with RHInterchange, may one day greatly facilitate commodity ordering and procurement. The Caucus, as another example, influenced the UN Commission on Life Saving Commodities for Women and Children to include family planning broadly and three contraceptives in particular among 13 essential commodities. The Task Force was similarly influential in having its three priority commodities included within the UN Commission’s list. Eventually such efforts will have impact all the way to country level. However, though country-level Coalition membership has increased, within the current structure the RHSC has yet to build much upon this growing reach. To do so, RHSC will need to re-strategize and re-focus to place some needed priority on country-level engagement and participation. The internal structures of the RHSC were set out in the 2009 Terms of Reference. The structural core of the RHSC – three Working Groups supported by designated Secretariat staff – remains largely what it was when the Coalition started: the seed bed for connections among expert staff in well-funded organizations. Clearly there has been some evolution and adaptation since that time. For example, the Caucus emerged in response to member- generated interest in focusing on under-utilized methods. Its continuing existence testifies to RHSC’s adaptability as well as efforts to cross-connect the Working Groups. Similarly, the Task Force on Maternal Health Supplies also emerged in response to member-expressed need, most visibly at the Access for All Conference in Addis. The Working Groups have been operationally idiosyncratic with varying effectiveness. Each has its own unique character of participants. For example, the Systems Strengthening Working Group (SSWG) continues as the work space for the original core of logistics experts, many from or affiliated with USAID, UNFPA, and JSI. The Market Development Approach Working Group (MDAWG) has been the most explorative of the groups, reaching out most recently to learn about promising new products and possibly influencing how they may come to market. The RMAWG comprised of advocates has been consistently most concerned with moving beyond commodities to change policies and have an impact at country level. 42 Evaluation of the Reproductive Health Supplies Coalition All three Working Groups seem have gone through periods of high involvement as well as stagnation. This is not unrelated to the availability of funds to actually implement activities. For the RMAWG, Project RMA initially and Advance Family Planning subsequently provided funding to implement advocacy efforts, notably at country level. Similarly, Deliver’s active presence in the Systems Strengthening Working Group provided opportunity for in-kind contributions from this well-funded Project. Without such supplementary support, for amounts and time frames beyond what the Innovation Fund could provide, the Working Groups may have long since faltered. After several years under the same leaders, the three Working Groups similarly experienced some loss of momentum and energy. This is being addressed by transitioning Working Group leadership positions as well as repositioning support from the Secretariat. Two of the three Working Groups are now operating under new leaders, with the third soon to change as well. Secretariat staff members are attentive to providing what the Working Groups and their leaders need. In the interest of more closely aligning activities with Working Group strategies, the Secretariat has recently encouraged that visioning exercises be conducted within the different Groups. Support from professional facilitators to plan and run RHSC meetings could add value to Working Groups as well as other Coalition convenings. The Innovation Fund began in 2008 with support from the Gates Foundation, which allocated $2.0 million to support RHSC Working Group generated studies and projects. To date, the review committee (comprised of selected Executive Committee, Secretariat, and PATH staff members) has approved 16 awards. With funding in the range of $50,000 to $200,000, successful applications from the three Working Groups included studies of new areas for commodity security advancement, such as pre-qualification; applications of specific logistics and market analysis tools within country settings, such as applying Total Market Initiative, a market segmentation tool, in Honduras and Madagascar; and production of the documentary Empty Handed on the importance of contraceptive logistics. A summary assessment of results at country level is at Annex 9 For the most part, the Innovation Fund has not been truly innovative nor has it approved proposals from far beyond the usual suspects. As several respondents expressed concerns about the Innovation fund, it has been misnamed and needs to be renamed. However, it has had the useful function of focusing Working Groups on particular subjects of interest and providing some funding to cover the costs of implementation. Without such supplementary funding, the Working Groups may have lost impetus and accomplished far less than otherwise. The Innovation Fund is further considered in the next section of this report, which considers partner-led and pro-active dimensions of the RHSC. 43 Evaluation of the Reproductive Health Supplies Coalition The RHSC builds on member interest in and contributions (in kind or through leveraged funding) that members may make to improve FP/RH commodity logistics and security. This “coalition of the willing” has built an organizational architecture that facilitates “leading from behind” or “leading from within”. However, the RHSC is not simply the sum of its membership structure, an accumulation of partner-led initiatives. While the RHSC Secretariat serves an increasingly diverse membership, over time it has also become selectively proactive, moving towards and sometimes advancing entirely new opportunities for commodity security. How this has come about and the balance maintained between member-led and proactive is pursued in this section. Many Coalition initiatives exemplify a member-driven and directed RHSC. The Working Group structure, as codified in the 2009 Terms of Reference, builds upon the RHSC’s commitment “to be participant driven in its strategy, implementation, and governance…and to working collaboratively and effectively to…resolve supply problems and ensure long-term RH supply”. Key informants often referred to Working Groups as the “engine that drives the Coalition”. The following provide specifics on how, when, and with what results partner-led has come to characterize the RHSC. “Coalition of the willing” well defines the external as well as the internal operations of the RHSC. Members self-select into the RHSC, whose only requirement is that organizations be duly registered. (Individuals are not allowed to join.) Joining the RHSC implies some alignment with its overall mission and vision. As part of orientation, new members are asked to affiliate with one of the three Working Groups. However, there are no expectations or obligations as to how much involvement there will be or what this will entail. As several key informants noted, new members often only participate in a single Working Group meeting, usually one held in conjunction within an annual meeting. Within the Working Groups and other member structures, clearly some individuals and organizations participate more than others. The discussions, deliberations, and decisions about what to work on are often useful for advancing some particular aspect of FP/RH. However, who participates and to what extent is a matter of willingness, interest, and also opportunity. For some, the structure works well. Those more distanced from or practically challenged by the level and substance of Working Groups, participation in these internal structures is not likely to happen. “Coalition of the willing” effectively defines an inner coalition comprised of those who are most able to be and already are well-connected. As one key informant stated,a coalition of the willing…(where) the work gets done by all the membership; that's all right up to a point, but you know, it’s not my day job.” 8. To what extent has RHSC been responsive to partner-led initiatives or proactive towards new opportunities? 44 Evaluation of the Reproductive Health Supplies Coalition Working Group projects and studies receiving support from the Innovation Fund were clearly partner-led. Sometimes these involved collaboration among member organizations, often for activities that would not otherwise have been implemented. The Secretariat’s internal review of the Innovation Fund, conducted in late 2010 by surveying members, forthrightly exposed several shortcomings and weaknesses. Though the Fund served some useful purposes, a number of administrative as well as substantive improvements were indicated. Among these were needs for greater clarity about how the Fund furthers the RHSC Strategy and Working Group operations, need for improved review criteria and processes, and need for follow-up or further application once a project or study is completed. In addition, no awards were made for proposals emanating from country-level. To correct for this, funds remaining have been set aside for applications from in-country organizations. To assist with proposal development, the RHSC has given a small amount of support to Population Action International. The evaluators agree with these conclusions of the internal review and the effort to make some country-level awards. The Innovation Fund has also fallen short on what could have been a hallmark of these awards: fostering collaboration among member organizations. Almost all awards involved projects and studies conducted by a single implementing organization. Left to their own and despite the connections that occur in the Working Groups, some organizations may seek to further their own interests and agendas than to more broadly contribute to the field. Proposal criteria that prioritize connections and collaborations across member organizations, including those at county-level, should receive greater weight. What should happen to the Innovation Fund going forward? The Innovation Fund has already supported 16 projects and studies of interest to Working Group members and is also being replicated within the LAC Forum. The small amount remaining in the Fund is currently being directed towards in-country organizations, a useful effort to support country participation. As a respondent to the member survey remarked, “(Being a Coalition member provided) opportunity to compete for funding though we never got any, but is another chance to learn from our shortcomings, as we look for funds to implement our activities“. Going forward, without a replenished Fund, Working Groups will very likely lose momentum. However, as several key informants observed, for the most part awards have largely not been innovative. Some suggested renaming or reorienting the Fund. Perhaps a Country Catalyst Fund, with weight placed on collaborative work at country level, would be a useful adjustment. From early on, RHSC members were clearly empowered to organize time and effort around supply initiatives and commodity security concerns. Annual Meetings often provide opportunities to involve partners in leading various sessions. For the most part, however, partner-led occurred within the Working Groups. In addition, when interest emerged and coalesced around particular subjects needing sustained attention and effort, members themselves created new structures outside the Working Groups. The Secretariat, recognizing the importance of such partner-led initiatives, quickly organized staff resources to provide some administrative and backup support for such nascent structures. Discussions and deliberations among members are the seedbeds for such partner-formulated structures. The Caucus on New and Underutilized Reproductive Health Technologies and the Task Force on Maternal Health Supplies are two member-initiated structures that took shape in this way and continue to work under the umbrella of the RHSC. 45 Evaluation of the Reproductive Health Supplies Coalition As early as 2006, around the same time that the RHSC itself was taking shape, two Coalition members initiated discussions for what eventually became the Caucus on New and Underused Methods. As one key informant explained, “(the Caucus) didn’t fit anywhere really . there’s a bit of resource mobilisation, there’s a bit market development, there was a bit of SSWG, so it sort of went across.” Another who was involved from the beginning described how the Caucus came to be accepted: “The (RHSC) was flexible enough to say ‘OK here is a need and some of our members think this is important’…we didn’t really ask for permission. We said we’re going to do this but what we want is for the (RHSC) to pay attention to us. So this is part of the membership and we framed it as a membership initiated activity…So you begin to see how these new themes come in because of a decision to open things up.” Attention for maternal health supplies actually arose first within the Caucus and then gained momentum at the 2011 Annual RHSC Meeting. A key informant, who participated in a roundtable lunch in Addis on maternal health supplies, described what happened in the aftermath: “There was all this real interest and obvious momentum, so we decided to form a task force that would talk about how the Coalition could add value to efforts to address maternal health supplies”. In follow-on discussions, the Task Force became highly focused around three commodities of particular interest: oxytocin, Misoprostal, and magnesium sulphate. The emergence and the willingness to embrace the Caucus on Underutilized Methods and Maternal Health Task Force exemplify a Secretariat that is “leading from behind”, as both staff and members describe it. In recent years, however, this approach has been complemented at times by leadership from the uppermost levels of the RHSC. The organization’s maturity combined with its growing stature within FP/RH, and development more broadly, has made this possible. In subsequent revisions and updates of the RHSC Strategy and Terms of Reference, which the evaluators recommend, this emerging role of the RHSC will need to be clearly considered and articulated. Maintaining balance and trust within an organization that is de facto built on consensus among the membership is important. In the following are two instances where the RHSC effectively exercised leadership on behalf of its members and to advance the field. The convening power and neutrality of the RHSC opened the way to developing relations with the manufacturers of FP/RH commodities. That this has occurred independently from the donors and governments directly involved in procuring these products has had distinct advantages. Encouraged by previous Executive Committee Chairpersons, the RHSC director met with various contraceptive manufacturers. Being better informed about products and perspectives proved mutually beneficial. Though some members initially opposed the idea, eventually several large pharmaceutical companies as well as generic manufacturers involved in FP/RH commodities became RHSC members. To date, their engagement has not been at the level of active Working Group participation. However, the neutral space that the RHSC provides has brought into the open a number of issues. Perhaps most notable among these is that price negotiations occur hand-in-hand with deliberations about supply predictability, volume, and sustainability. 46 Evaluation of the Reproductive Health Supplies Coalition The Campaign to focus global attention on meeting unmet needs of 100 million women for contraception by 2015 started as an idea at the highest levels of the RHSC. As related by both the current Chair and the RHSC Director, Hand-to-Hand emerged as an opportunity to focus global attention on an important issue. As such, it has already had the impact of galvanizing donor attention around the upcoming Family Planning Summit to reinvigorate funding for family planning towards a revised metric of 120 million by 2020. Among members, however, it is not clear whether Hand-to-Hand has been a hallmark success or a not-fully- thought-through initiative that will disappear as quickly as it came. The higher level perspective within RHSC is that “Hand to Hand is one where the community comes together and says we will work towards this, we will work towards this goal.” As further elaborated by one of the conceptualizers, there was also strategic sense behind it: “(The) Hand-to-Hand campaign has been a really strong mobilising force for the Coalition and has given it a sense of purpose …(The RHSC does not have) a kind of really strong uniting purpose…Lots of very interesting people, lots of individual commitments to the issues, but nothing that brought us together to deliver something tangible. So that for me was the purpose behind Hand to Hand.” Some advocates within the RHSC however see Hand-to-Hand as something that lacks strategy, not an approach that imparts it. The following extracts from key informant interviews convey this clearly: “(Hand-to-Hand) happened so quickly…there were a lot of questions: Toward what end? What’s the goal here? What are we attempting to achieve? … we’ve discussed within the RMA Group of how do we orient our strategy, our work, in support of Hand-to-Hand, but … it’s not really clear what the Coalition is seeking to achieve…in the RMA Group, where there is interest and willingness, but not certain that this has staying power…misunderstanding of advocacy as PR, as communications…(when) it’s much deeper than that.” “I wasn’t really that involved in it at all, but it just seemed like it was a campaign that had good intentions, … just hired a private firm to basically create a campaign… which isn’t really how we do campaigns because there’s a lot of legwork and a lot of prep work and getting partners on board.” Like any organization, the RHSC has strengths and weaknesses. For example, members with separate funding well aligned with a particular Working Group are often the most active participants, usually the central office staff of large international organizations. In this way, the Systems Strengthening and Resources Mobilization and Awareness Working Groups were historically better endowed than the Market Development Approaches Working Group. In part the Innovation Fund was created to address this imbalance. A related but different imbalance apparent across all Working Groups is the lack of country- level participation and involvement. Even when Working Groups, such as the RMA and MDA, have engaged in country-level activities, lack of country representation continued. The recent election of the country representative, himself a staff member of an international organization separately funded for an advocacy project, to co-chair the RMAWG is a notable exception and worthwhile step in the right direction. 47 Evaluation of the Reproductive Health Supplies Coalition While key informants indicated awareness of such imbalances, members have not as yet been actively engaged to find solutions. This is likely to be a leadership role for the Secretariat to undertake, perhaps in conjunction with revising the Strategic Plan. Some key informants, donors as well as implementing organizations, did express that the Secretariat should exercise more strategic leadership, including taking positions on behalf of the Coalition with regard to particular issues. Instead of providing background support, for example, to the upcoming FP Summit, the RHSC could put forth a position on the Summit itself. This would clearly be a diplomatic and delicate task. At present, means for gathering consensus among members are not clear nor would this likely occur without some dissension among the ranks. The foundational documents, the 2007-2015 Strategic Plan and the 2009 Terms of Reference, firmly fix the RHSC as a members-driven organization. The related mantras have been that the Working Groups are the engines of the Coalition and that the Secretariat leads from behind. However, from the examples above, it is clear that these perspectives are no longer entirely accurate. As the RHSC matures, it will be useful to define and describe its structure and operations in a way that reflects how it does and will likely continue to operate. Bottom-up, member-driven proactivity as well as top-down leadership will both be important for the RHSC to succeed in the coming years. It is important that members understand and embrace both approaches. 48 Evaluation of the Reproductive Health Supplies Coalition In 2004, “No Product, No Program” went mainstream when the Interim Working Group on Reproductive Health Commodity Security reformulated as the Reproductive Health Supplies Coalition. Today, nearly 200 organizations are members of this Coalition. In this section, we assess the readiness and capacity of the RHSC to sustain this movement and continue advances for FP/RH commodity security. In recent years, many advances have occurred within contraceptive and reproductive health commodity security, a number directly attributable to the RHSC. However, when one key informant, a well-placed donor representative, considered future pathways for the RHSC, he raised an important question and recalled a seminal study commissioned earlier by the Gates Foundation: “Why don’t we have country-level impact? We still have that study from way back when, Mercer, that showed end-to-end supply chain: where are the problems that ultimately manifest themselves in stock-outs? If you attack the supply chain at country level, you make about, I think they said, 70% difference. If you attack the problem at higher, global level, which is where we focused: about 30% difference. Let’s be honest, that’s what we did – Pledge Guarantee, AccessRH, CARhs. Fine! Done that! Let’s get to the 70% now, and that would require turning the Coalition on its head!” Since the 2005 RHSC Meeting in Seattle, where the Mercer Study as well as country mapping were both presented, country-level involvement has been a recurrent theme for improving FP/RH commodity security. This perspective continues to remain essential and is central within this consideration of RHSC’s potential for continuing to make advances for FP/RH commodity security in the future. Key informants were nearly unanimous about need for the RHSC to focus more on country- level impact. As one former high-level director of a large donor agency stated of the RHSC, “They have to get more involved, and (its Director) readily admits this; they have to get more involved in the South. Contraceptive security is never going to be realized unless countries take more ownership in contraceptives, in their purchase, and their distribution.” However in terms of what should actually be done, key informants were much more divided. Opinions ranged from doing more to keep country pages on the RHSC website current to becoming directly involved with members in supporting country-level activities and interventions. As shown below, a number of respondents independently concluded that the RHSC should focus within a select number of countries: “The Coalition can bring value … and maybe one can redefine (its) role in a way (as) technical assistant/ advisor/consultant… which they could send to countries to help to solve problems (by) consulting, (providing) expertise, knowledge, technical assistance…if you think about supply in a broader sense beside products, you even can come up with supply of 9. Does RHSC seem poised and/or have potential for improving and increasing contraceptive security in the future? 49 Evaluation of the Reproductive Health Supplies Coalition information, supply of advice, supply of consultancy work… select maybe one or two or maybe three pilot countries… (and) get to a more, let’s say, cross-country way of learning.” “I wonder what the Coalition could perhaps have tried to have done to become relevant in a small selection of countries… a strategic opportunity for the Coalition to say we’re going to try and make that happen in these five countries… to demonstrate that the Coalition’s in a position to really make a difference. That the Coalition, that global thing, in these five countries can do that.” “at the country level… you (could) have a country or a region representative from the Secretariat going around specific countries engaging people, meeting with them regularly, you know kind of encouraging their participation, disseminating tools and resources, all of that… if it were a focus country to encourage southern member participation in the Coalition then I could see that being more relevant than the way (the RHSC is) currently set up.” A respondent to the electronic survey of members, a staff member of a private foundation, added this comment: “I am fully supportive of the Coalition being active in 2-4 countries and testing whether they can build a comparative advantage that serves the field”. Another respondent, this time from a bilateral donor agency, was more expansive on the same theme: “The early success of the RHSC has been focused on improved coordination,raising commitment,and the development of innovative supply approaches at the HQ level.There is an opportunity to amplify the impact of the RHSC by strengthening coordination, commitment, and innovation at the country level.This is happening already in some countries, but the RHSC could make these efforts more intentional and expand effective approaches to more countries.” In fairness, not all respondents saw country-level as a present strength or future emphasis for the RHSC. A dialogue within one large INGO between key informants made this point very clearly: Respondent 1: Could (the RHSC) do something that brought more country people onto the same page? Yes. Would it necessarily happen in-country? I don’t think so. I think the Coalition’s power lies outside the country and it may be regional or it may be global… Respondent 2: And I go back to the Contraceptive Security indicators. . Who is on it? By getting that information out in a decipherable way to the Coalition members, the countries, maybe it helps to call attention to what is or what isn’t happening. It is a data point that helps to track and hold them accountable by just sharing that information. But beyond that I don’t see the Coalition doing anything. Respondent 1: Yeah, I don’t see it at the country level. In general, however, most key informants and member survey respondents were favorable to the RHSC being more actively involved at country level. Of 65 who completed the electronic survey, 55 agreed with the statement that “the Coalition should engage more at country level to expand FP and commodity security.” Interestingly, of these only 4 of the 9 respondents from INGOs agreed. The Coalition has grown to nearly 200 member organizations. Building on an original core of some 15 organizations, the RHSC is currently increasing by an average of 10 new members 50 Evaluation of the Reproductive Health Supplies Coalition a month. Having surpassed the rolls of the IPPF, the RHSC is likely the largest extant body able to call on organizations to join efforts for improving FP/RH. The Director put this expansion in perspective for considering RHSC’s future: “Now when you have almost 180 members (in March 2012) and you are really probably the largest single partnership family planning organization in the world, then in a way it kind of. it demands more of you and there are responsibilities that come from having that. And do you say, ‘Well, that’s fine’ and I ignore that and still keep on commodities, or do you kind of use commodities as a leverage to be able to help you address the broader issues that do touch upon issues important to the wider community? It is a good question and it’s one that we’ve discussed a lot.” Increasingly well regarded and widely recognized, RHSC now includes a wide array of stakeholders. In addition to the core INGOs and donors, the membership includes growing numbers of multinational pharmaceutical companies; generic manufacturers; international organizations involved in related health and development issues; social marketing organizations; government officials; and in-country civil society organizations. Membership however does not necessarily mean active involvement or participation. As a SSWG member observed during its May 2012 meeting, service delivery INGOs were not at the table. While service delivery organizations sometimes participate in the SSWG, those who regularly attend from PSI, MSI, or IPPF/ICON represent the commodity units of those organizations. Others who could notably add to Working Group discussions are civil society organizations and manufacturers. To its credit, the recent MDAWG meeting in Palo Alto did reach out to include some supply industry representatives. Operating in conjunction with the RMA Working Group from 2006 was Project RMA, led by Population Action International and funded by the Gates Foundation. Project RMA was an international advocacy initiative at global, regional, and country level. However, though Project RMA had notable achievements within a few countries, the weight of this Project ultimately proved deleterious to the RMAWG, particularly those members who did not have access to these funds. As some key informants reported, the end of Project RMA led from 2010 onwards to revitalization of the RMA Working Group. A contributing factor has been a similar project, Advance Family Planning (AFP) with funding from several donors. AFP appears to have successfully used the RMAWG as a platform from which to engage civil society organizations to advocate for commodity security in several African countries. If a current request for renewed funding is successful, the AFP and the RMAWG will continue active involvement at country level in coming years. With membership likely to continue increasing, the future of the RHSC depends on three inter-related factors: member commitment and involvement; funding to support member initiatives, including those of the Working Groups, the Secretariat, and its value-added contributions; a Strategic Plan that is re-focused and revised, likely moving towards having impact further down supply chain. A perennial issue within the RHSC concerns how broad should its substantive reach be. Some members expressed concern that moving beyond contraceptive supplies could be detrimental to RHSC support for FP. However, others have encouraged and embraced this in such areas as maternal health supplies. For example, the Task Force on Maternal Supplies, which emerged from member interest, was not embraced by all. As one donor long associated with the Coalition noted, the RHSC “is broadening out, you know, to do maternal health supplies and to do supplies during emergencies, etc. I think that it’s good that they’re 51 Evaluation of the Reproductive Health Supplies Coalition doing these things, but again (it’s important) to keep everything in balance towards the priority (of contraceptive commodities).” Another donor made a similar observation, but put this into an even broader context: “(The RHSC has) been very flexible in adapting to opportunities that arise and structuring themselves and the members in a way that delivers and meets the needs of new constituency groups and new opportunities. I guess the question for all of us: when new things come along is: what drops off?.Does it…strengthen its broader reproductive health mandate? Does it move into some of the other stuff around demand, crystallization around operational planning and (service) delivery?” Some key informants expressed concern that moving beyond contraceptive supplies could be detrimental to RHSC support for FP. However, others have encouraged and embraced broader involvement in such areas as maternal health supplies. As one long-time member of the MDA Working Group reflected, “It is all procurement. And as contraceptive prevalence goes up and governments will move into a range of areas, there is no question that we have to think about (other commodities) this as an entire market.” Responses by key informants to broader extension – beyond commodities to advocacy and even service delivery – were more mixed. The RMA Working Group, for example, supports some country-level advocacy, with technical and financial support from Advance Family Planning promoting, government commitments to and procurements of contraceptives. Though some key informants argued that demand generation was also important, for many, including donors, this would be beyond the purview of RMAWG and AFP as well as the RHSC itself. Similarly, for service delivery, though having service provision organizations in the Coalition is important, extending RHSC in that area would likely be limited. The breadth of RHSC can be reconsidered within the update of its Strategic Plan, However, with form following function corresponding changes may also be needed for the Secretariat, adding staff for example who have policy and possibly service delivery expertise. Most key informants wanted to maintain a core focus on commodities; however extending to areas beyond contraceptives was widely if not totally accepted by all. Similarly, member survey respondents who favored greater RHSC involvement at country level were evenly distributed among the areas where they thought the RHSC should engage. Selecting among three of seven choices, most respondents (55%) opted for resource mobilization and financing of RH commodities. However, equal numbers (49%) also selected increasing demand for FP/RH services/supplies and supply chain strengthening. For some interviewed, the concern that started more than 10 years ago to get the field more fully focused on commodities has now come full circle. As one of the originators of this movement stated: “Social change is not going to happen by putting out a bunch of contraceptives only… in the past 20 years people have shied away from that aspect of what we always used to called IEC and what was always part of the family planning programming was changing people’s minds – one woman, one couple, one community, one district and one nation –and nobody does that anymore.” This same topic was also raised during the recent SSWG discussion on future plans and directions. In the process, the relative lack of service delivery organizations among RHSC members was noted. 52 Evaluation of the Reproductive Health Supplies Coalition Since 2006, the RHSC has maintained a structure of Working Groups within which members work to advance FP/RH commodity security. The RHSC website provides a platform to communicate not only the results of that work but also broader developments within FP/RH commodity security. To maintain the website, the RHSC relies on inputs from members and outsources technical updates to consultants. Though some members remarked that the website lacked depth and was in many areas not current, it remains a good repository of what has occurred within the RHSC structure. As a portal to resources, data bases, and logistic system learning opportunities, the website is well positioned and appears well used. The Secretariat, again with members’ input, also produces a monthly newsletter, a useful update on relevant achievements and events occurring across the field. Clearly with resources and attention, the website may be improved. A solid base to build upon, the website could also considerably expand. More attention could be given, for example, to making the website a more interactive forum for information exchange. The development of Learning and Professional Training Opportunities (LAPTOP), a tool to connect supply chain managers with web-based courses on procurement, logistics, contracts, and other related operations, is a step in the right direction. Similarly, the Secretariat could use the website to bring focus to issues of particular importance for FP/RH commodity security. As one example, the RHSC could monitor funds pledged during the FP Summit. Posting expenditure information on the RHSC website would help turn this July event from the metaphorical wedding into the lasting marriage that some donors hope for. Similarly, should the RHSC place priority on outcomes within some countries, these could be highlighted within revitalized attention to selected country pages. As its rapidly growing membership indicates, the Coalition is clearly poised to improve and increase contraceptive security, both as a field of knowledge and through practical applications to improve systems at various levels. However, it does not currently have the bandwidth or depth among Secretariat staff to do so, nor does its dated strategy provide specific pathways for moving further along the supply chain. A woman in Ethiopia who receives Implanon becomes an ultimate beneficiary of price reductions that the Coalition helped to broker. However, such direct line connections between an RHSC contribution and a result remain rare. The challenge for the future, and one that was embraced by the majority of key informants, is: How can the RHSC position itself to have greater impact in the coming years? Already being discussed among members, the evaluators suggest that revisiting and revising the RHSC strategy would usefully concentrate attention upon this important question. The ethos of being member-led can and could be balanced by a Strategic Plan that orients Working Groups towards promising lines for advancing FP/RH commodity security. Instead of the current RHSC Strategic Plan that encompasses and coordinates within what member organizations already do, the expanded member-base itself could be involved in revising the strategy towards higher level goals of what can be collectively accomplished. In responding to the next and last question (Section 10), the evaluators highlight some additional ideas and recommendations for future consideration that emerged during this evaluation. 53 Evaluation of the Reproductive Health Supplies Coalition There is clearly some pressure within the Coalition to widen the focus of commodity security. The evaluation team recommends retaining the current focus on RH. Currently this includes a comprehensive array of family planning products as well as a more focused set of maternal health supplies. Though the Coalition has contributed to repositioning the supplies agenda, there is still a long way to go. Contraceptive commodity security achievements in particular could well be compromised without continued Coalition attention to family planning. However, encouraging coordinated action to secure a slightly wider range of related health supplies may in time become politically practical as well as practically achievable. The question of whether the Coalition should operate more directly at country level was much discussed in the course of this evaluation. Many Coalition members recognize that commodity security remains weak in many countries and encourage greater RHSC involvement and efforts. The Coalition accepts this as a shared objective requiring greater collaboration to achieve. Key informants proposed various options for RHSC engagement further along the supply chain, from supporting mini-coalitions in some countries to taking concerted action in a small number of selected focus countries. However, beyond broad agreement for doing more to engage at country level, there was no consensus. The evaluation team recommends that the RHSC maintain its global-level focus, continues to reach toward countries through existing regional forums, and considers concentrated engagement in a select number of countries where value would be added. The evaluation team also recommends increasing country-level ownership of the RHSC by making the Coalition more relevant to country needs. The Strategic Plan states: “While international expertise can play a critical role in recognizing weaknesses and identifying technical remedies, the search for long-term solutions will only succeed when countries themselves define the roles of their partners, including the Coalition, and set their own supply agenda. The Coalition will, therefore, reinforce the principles of country leadership and 10. Beyond serving as an umbrella for collaboration, what alternative models or approaches could and should the RHSC adopt that are needed in the coming years to advance RH commodity security globally, regionally, and at country level? 54 Evaluation of the Reproductive Health Supplies Coalition ownership in all country-level RH supply security activities”. This remains relevant for the work of the Coalition but in practice this is not how the RHSC operates. The Coalition should consider what it can do at global level to help countries. Could country- level members identify common critical issues in commodity security to which the Coalition can respond? Does the RHSC simply need to ask its country level members: ”What could we do to help you?” This would not be a straightforward process. At the moment the voices of southern partners are not often heard within the Coalition. With the exception of one co-chair, Global South engagement with the Working Groups is virtually non-existent. As currently structured, this means they miss out on much of the agenda setting. The team experienced difficulties involving Global South members in this evaluation, not unlike those that the Coalition faces on a regular basis (e.g. poor phone lines, lack of internet connectivity). However, like Coalition members generally, country-level contributors to this evaluation, both key informants and member survey respondents, clearly expressed value in being members. Some also provided useful input on why Global South engagement is limited. Cost is a big issue, not only to attend meetings, but even joining conference calls is constrained by cost as well as time. Also limited is knowledge of RHSC tools and approaches and their potential benefits for those who would immediately benefit, such as Ministries of Health. Strategies to increase the participation of Global South members could include: Developing different forms of engagement with the Working Groups, which currently rely on members attending and actively contributing. For example, members could undertake consultative exercises about new priorities, through email or by reaching out through their local country offices. Having one or two country members on the Executive Committee in revolving posts. Forming a country Working Group which, borrowing from the Caucus practice, informs the work of other WGs. Funding participation by southern partners in Working Group meetings (i.e. paying for phone calls, selected travel). Ideally, each WG would have a budget to cover such expenses on a regular basis. Funding attendance by more strategically selected southern partners at the Annual Meetings and working out how to better engage them, and subsequently sustain that engagement. Developing communication approaches to reach existing ‘silent’ southern members. Developing mechanisms that encourage/promote cross-country exchanges and interactions on commodities to create country level momentum. Supporting more links between northern and southern organizations, as has occurred in some countries through the RMA Working Group. This could be applied across the RHSC more broadly. The Strategic Plan states that ‘the Coalition relies on the contributions and mutual accountability of its members.’ In 2007, the Gates Foundation was the only donor providing core support. Some additional funding was subsequently made available by BMGF and managed through the Innovation Fund. At a total of $2m this was enough to initiate ideas, but not enough to ensure longevity. Realistically the RHSC will remain largely dependent on the contribution of members, not a bad arrangement in many ways. Volunteerism increases members’ stake in making the RHSC work, focuses the Secretariat on the need to provide added value to members, and encourages everyone to prioritize. It is also a good barometer 55 Evaluation of the Reproductive Health Supplies Coalition of how well the Coalition is doing: volunteers can and do walk away if they do not get the returns they want. Despite such advantages, some Working Groups have felt disadvantaged. At different times and to varying extents, RMAWG and the MDAWG members have not had flexible funding to support Coalition activities. The evaluation team recommends that consideration be given to providing selective financial support to Working Group functions and operations. Different from Innovation Fund support, this suggested supplementary funding would support Working Groups to function effectively. It could be allocated as budgets for each Working Group or controlled centrally on a case by case basis. Ensuring this did not act as a disincentive for members to invest would be important. Such supplementary funds could also be usefully tapped to facilitate and engage southern member participation. The other problem with lack of funding to the Coalition is that some initiatives wither before being fully and fairly tested. The PGH is a case in point. The evaluation team recommends that consideration be given to housing some of the best but most vulnerable initiatives in the Secretariat, starting with the PGH. The PGH has the added advantage that it could potentially be self-financing and eventually generate funds which could perhaps be used to support other initiatives. The preparedness of donors to use PATH as a pass-through arrangement, and the fact that this arrangement would only apply to home grown initiatives may remove some of the potential problems identified below with ‘ready formed’ initiatives. The evaluation team also recommend that the Innovation Fund be renewed, renamed, and reoriented. The Innovation Fund supported some useful work, has helped pull some Coalition members together, and generated momentum for a few Coalition-germinated initiatives. However, Innovation Fund grants were won by agencies which not only put forth good ideas but which were also good at developing proposals. As a result, small southern NGOs, inexperienced in producing international quality grant applications, were not successful despite best efforts to include them. Also many of the ideas funded by the Innovation Fund were not actually very innovative, even though they were useful to have done. A renewed and expanded funding stream should be instituted to encompass a wider range of initiatives, tailored to different constituencies of the Coalition. This would place priority on collaborative projects and activities and could include: Innovation Fund for truly new ideas Dissemination / Continuation Fund for in-country field testing and further development of existing tools Country Catalyst Fund to support advocacy and other initiatives, e.g., provide support around upcoming events, important country-level decision points, or potential supply system or service delivery breakthroughs. This Fund would particularly target country- level activities and organisations. Building on support that PAI is currently providing to local NGOs, including similar upfront support for proposal development would also be useful . There is potential for the Coalition to take a stronger role in leading on RH commodity security. That it has many and possibly all the key members for commodity security at global level gives RHSC a potential voice and power not fully exploited to date. Caucus efforts to have FP commodities included by the UN Commission on Life Saving Commodities for Women and Children, though a response to an omission, is an example of what can be done. There may be potential for the Coalition as a body to be more proactive in identifying key issues, taking a clear public position, and moving the agenda forward. The HandtoHand 56 Evaluation of the Reproductive Health Supplies Coalition Campaign had the potential to do this but has been overtaken by the Family Planning Summit. In the future, a stronger leadership role for the Director may be useful. At present, the Director is essentially the lynchpin of the membership network, the Coalition’s ‘front man’, someone who moves things along, often leading from the middle rather than the front. This role was appropriate in the earlier days of the network when the key task was to bring together a disparate, sometimes cantankerous membership; it remains valuable for maintaining neutrality. But there may now be scope for being more proactive on some strategic issues. This may imply more of a mandated role, with the annual meetings used to build member consensus around issues and set strategic direction. UNFPA, with its Global Programme to Enhance Reproductive Health Commodity Security (GPRHSC) is a key member of the Coalition. The Programme is unique for many reasons not least of all its scale, country reach, emphasis on both commodity supply and capacity building, flexibility, and ambitious goals very much in line with what the RHSC is trying to achieve. The Programme is now going through a process of revision for new donor funding. The close relationship between the Coalition and UNFPA has been important for both parties, and UNFPA staff are active within RHSC operations in different ways, e.g. Executive Committee, Working Groups, some of the Innovation Fund initiatives, and in the delivery of AccessRH. However relations between Coalition members and Programme implementers have been more tenuous. As a result they do not always work together at country level in the most effective way. The evaluation team recommends that the RHSC and UNFPA give special attention to how the GPRHSC and members programmes can be mutually enhanced, both at global and country level. This should involve the RHSC having a more formal link with the Programme (perhaps a seat on the proposed consultative body) and some strategizing on how lower level links can be created and operations of the GPRHSC enhanced. Encompassing regional forums into the Coalition is both pragmatic and promising. Placement of a regional advisor under the Hewlett grant to SECONAF is an opportunity to influence developments in that region and learn from that experience. The SECONAF experience should be tracked for lessons learned for having impact at country level. It is not yet clear however whether regional approaches will generate country level impact; moreover no metrics currently exist to monitor this for LAC, an omission in the MEF which should be addressed if politically possible (and certainly should be addressed for SECONAF). The evaluation team recommends that the Coalition continue support for the two regional forums. However, the Coalition should remain cautious about regional approaches generally, for which there are many more failures than successes. Groupings of countries (like-minded, same stage, or self-selected to learn from each other) may be a better way to proceed and this could be encouraged within the regional forums. As discussed previously, the current Strategy (2007 – 2015) has served the Coalition well but is now somewhat out-of-date. The evaluators recommend that the Strategic Plan be refreshed and revised to reflect the expanded membership and in response to the suggestions above. An explicit theory of change would usefully contribute to both formulating an updated strategy and revisiting the MEF. 57 Evaluation of the Reproductive Health Supplies Coalition The strengths of the Coalition, as discussed in section 4.1, are its neutrality, the breadth and depth of its membership, its power as a convenor, the commitment of its members, and the effectiveness of its Secretariat. RHSC neutrality has been hard won and is its greatest asset. It is essential that this and its other strengths are preserved even as the RH landscape, in general and for commodity security in particular, evolves and new potential opportunities for the Coalition emerge. The evaluation team recommends that the Secretariat has its capacity increased, to both better service the existing needs of the Coalition and step up its leadership, as discussed above. The Director needs more time to lead more pro-actively. Increased technical capacity within the Secretariat would free the Director for higher level leadership and provide valuable support to the Coalition structures. Secretariat capacity is currently increasing with the addition of full time communications and M&E officers. The evaluators suggest that RHSC capacity needs be further considered as the landscape for RH generally and RH commodity security in particular continues to develop. The evaluation team recommends that the Executive Committee be reconfigured to permanently represent more member constituencies (e.g. manufacturers and country-level members, if practicable), and to bring on some independent strategic thinkers to act as short term advisors, with terms of perhaps 1 to 2 years. This would enable the Executive Committee to widen participation, and to be continuously challenged. Key informants both value and criticize RHSC Annual Meetings and other Coalition gatherings such as Working Group meetings. The evaluation team recommends that RHSC continue to conduct annual meetings and support other forums where members come together, but with some needed improvements. Annual meetings, for example, could be a useful forum to formally strategize within the Coalition as a whole. Similarly, these occasions could be used to build consensus among members around issues on which the RHSC would take leadership. Annual as well as other meetings also need to be used to engage countries more systematically. Expanded support for strategically selected participants as well as for follow up would provide added value. Holding annual meetings within strategically selected countries, as has occurred in Uganda and Ethiopia, should continue. These have proven useful venues to mobilise government support, show-case successes, and encourage members to lend support to future efforts. The RHSC needs to continue being strategic about where and with which members to undertake such investments. Given the tremendous need for FP/RH in francophone Africa countries, the October 2012 meeting in France seems another strategic venue selection. On a practical note, the Secretariat needs to upgrade how it organizes and facilitates its Annual Meetings as well as other meetings convened under the aegis of the RHSC. The evaluators suggest that the Secretariat identify a core group of highly skilled facilitators to plan and conduct such meetings. Methodologies, such as Open Space used with the 58 Evaluation of the Reproductive Health Supplies Coalition SECONAF contingent as part of the Addis annual meeting, should be expanded to include other new approaches. For example, Future Search and World Café have been successfully applied in a wide variety of settings to fully engage even very large groups, More importantly, by consulting with professional facilit
Looking for other reproductive health publications?
The Supplies Information Database (SID) is an online reference library with more than 2000 records on the status of reproductive health supplies. The library includes studies, assessments and other publications dating back to 1986, many of which are no longer available even in their country of origin. Explore the database here.