Evaluation of the Reproductive Health Supplies Coalition 2012

Publication date: 2012

305968 01 01 /// 20 June 2012 Evaluation of the Reproductive Health Supplies Coalition Commissioned by the Bill & Melinda Gates Foundation in association with USAID, DFID, and UNFPA Don Lauro Adrienne Chattoe-Brown (HLSP) 30th June 2012 Final report 305968///01/01 30 June 2012 Evaluation of the Reproductive Health Supplies Coalition Many people gave a great deal of time and thought to help with and support this evaluation. Representatives of the four donors funding the RHSC, DFID, the Gates Foundation, UNFPA, and USAID, developed a solid SOW for us to build on and then met with us in person to provide input for our evaluation work plan. At a later stage, they joined in a conference call to help us sift through preliminary findings. The staff members of RHSC, and particularly the Director, were patient with our presence in their office, the many hours of interviews we conducted there, and our frequent follow-up requests for additional documents and information. The members of the RHSC, who we interviewed as key informants, were open and willing to participate, much like their contributions to the Coalition over the years. We can honestly say that we learned something new about the Coalition from each and every person interviewed. The 65 members who responded to the electronic survey sent to all member organizations also gave us new perspectives and additional insight on the RHSC. Throughout this evaluation, we received a great deal of support from a variety of people for several tasks: conduct background research, transcribe interviews, develop and administer the members’ survey, and organize and format this report. We thank all who have participated in this evaluation and trust that they will find value and use for what we have learned about the RHSC and what we recommend for its future. Acknowledgements 305968///01/01 30 June 2012 Evaluation of the Reproductive Health Supplies Coalition Chapter Title Page Acknowledgements i Content ii Acronyms 1 1. Executive Summary 2 1.1 Introduction ________________________________________________________________________ 2 1.2 Achievements: Global, Regional, and Country Level ________________________________________ 3 1.3 RHSC Structures and Processes _______________________________________________________ 6 1.4 Future Directions and Recommendations for the Coalition ____________________________________ 8 2. Introduction 10 2.1 Scope of Work ____________________________________________________________________ 10 2.2 Organization of the Report ___________________________________________________________ 10 2.3 Methodology ______________________________________________________________________ 11 3. What contributions has RHSC made and what impact has it had on improving commodity security at global, regional, and country levels? 12 3.1 Overview _________________________________________________________________________ 12 3.2 Performance against the monitoring and evaluation framework _______________________________ 13 3.3 Country uptake and results of RHSC-inspired initiatives _____________________________________ 13 3.4 Impact at regional level ______________________________________________________________ 16 3.5 Influence on the global health agenda __________________________________________________ 16 3.6 Contribution of the Coalition __________________________________________________________ 16 4. Over and above partner organizations’ own accomplishments, what is RHSC’s comparative advantage, added-value, and value-for-money? 18 4.1 Comparative advantage _____________________________________________________________ 18 4.2 Added value ______________________________________________________________________ 20 4.3 Value for money ___________________________________________________________________ 22 5. To what extent is RHSC’s role adequately reflected in its strategy, theory of change and M&E framework? 26 5.1 Current strategy ___________________________________________________________________ 26 5.2 Theory of change __________________________________________________________________ 27 5.3 Monitoring and evaluation framework ___________________________________________________ 27 6. In what ways and to what effect has the structure and operation of RHSC evolved, particularly since 2009? 31 6.1 2009 Terms of Reference ____________________________________________________________ 31 6.2 Operational Structure of the RHSC _____________________________________________________ 33 6.3 RHSC Flexibility and Adaptations ______________________________________________________ 34 6.4 Support for Structures and Operations __________________________________________________ 35 6.5 Future Secretariat Needs ____________________________________________________________ 36 7. What works well and what needs to be improved or changed about RHSC’s internal structure Content 305968///01/01 30 June 2012 Evaluation of the Reproductive Health Supplies Coalition and external operations? 37 7.1 Strengths within RHSC Internal Operations ______________________________________________ 37 7.2 Weaknesses within RHSC Internal Operations ____________________________________________ 39 7.3 External Operations ________________________________________________________________ 40 7.4 Appropriateness of RHSC Structures and Processes _______________________________________ 41 8. To what extent has RHSC been responsive to partner-led initiatives or proactive towards new opportunities? 43 8.1 Partner-Led Initiatives within RHSC ____________________________________________________ 43 8.2 A Proactive RHSC _________________________________________________________________ 45 8.3 Coalition Upsides/Downsides _________________________________________________________ 46 9. Does RHSC seem poised and/or have potential for improving and increasing contraceptive security in the future? 48 9.1 Country Level: Origin and Destination __________________________________________________ 48 9.2 The RHSC Membership _____________________________________________________________ 49 9.3 Breadth and Depth of the RHSC _______________________________________________________ 50 9.4 Poised but Not Fully Positioned _______________________________________________________ 52 10. Beyond serving as an umbrella for collaboration, what alternative models or approaches could and should the RHSC adopt that are needed in the coming years to advance RH commodity security globally, regionally, and at country level? 53 10.1 Strategic ways forward ______________________________________________________________ 53 10.2 Operational issues _________________________________________________________________ 57 Annex 1 Statement of Work 59 Annex 2 Bibliography 66 Annex 3 RHSC Evaluation Workplan 69 Annex 4 Answers to Key Questions from RHSC Scope of Work 87 Annex 5 RHSC Member Survey 2012 113 Annex 6 Evaluation Methodology and Data Analysis 142 Annex 7 Summary of results against the MEF 145 Annex 8 Outcome of CARhs engagement. 149 Annex 9 Country relevant Innovation Fund grants 150 Annex 10 Contributions of the Supplies Coalition to Key Initiatives 152 Annex 11 Added Value of the Supplies Coalition 156 Annex 12 Presentation to Executive Committee, June 25th 2012 158 Annex 13 Reproductive Health Supplies Coalition. Monitoring Report (2009-2011) 169 1 Evaluation of the Reproductive Health Supplies Coalition BMGF Bill and Melinda Gates Foundation CARhs Coordinated Assistance for Reproductive Health Supplies DFID UK Department for International Development FP/RH Family Planning and Reproductive Health GFATM The Global Fund to Fight AIDS, Tuberculosis and Malaria GHP Global Health Partnership GPRHCS Global Programme to Enhance Reproductive Health Commodity Security H2H HANDtoHAND Campaign HLF High Level Forum IAI Implanon Access Initiative IF Innovation Fund JSI John Snow Inc. LAC Latin America, Caribbean M&E Monitoring and Evaluation MEF Monitoring and Evaluation Framework MDAWG Market Development Approaches Working Group NGO Non-governmental Organisation PAHO Pan American Health Organization PGH Pledge Guarantee for Health RH Reproductive Health RHCS Reproductive Health Commodity Security RHI Reproductive Health Interchange RHSC Reproductive Health Supplies Coalition RMAWG Resource Mobilisation and Awareness Working Group SECONAF Sécurité Contraceptive en Afrique Francophone SOW Scope of Work SSWG Systems Strengthening Working Group UN United Nations UNFPA UN Population Fund USAID United States Agency for International Development WAHO West African Health Organization WHO World Health Organisation Acronyms 2 Evaluation of the Reproductive Health Supplies Coalition 1.1 Introduction Julie Solo’s 2011 review captures how “no product, no program” moved from sideshow slogan to mainstream movement: “(in the late 1990s) a small group of committed and driven women…saw concern for supplies gradually slipping,…committed (themselves) to preventing that, and (formed) the Interim Working Group on Reproductive Health Commodity Security”. Reformulated in 2004 as the Reproductive Health Supply Coalition (RHSC or Coalition), 12 (mostly donor) organizations continued to coordinate work around family planning and reproductive health (FP/RH) commodity security. By 2006, with funding from the Bill and Melinda Gates Foundation, the RHSC had expanded to some 20 members, began supporting members from a Secretariat based in Brussels, organized into three Working Groups, and was overseen by an Executive Committee comprised of donor and other members. Presently, with this same structure basically intact, RHSC has grown to include 200 organizations and has originated and advanced a number of approaches to improve how FP/RH commodities are positioned, procured, and provided to those who wish to use them. Four major donors currently provide core support to the Coalition: the Gates Foundation; the United Kingdom’s Department for International Development (DFID); the United Nations Fund for Population Activities (UNFPA); and the United States Agency for International Development (USAID). These four donors commissioned this evaluation and identified two consultants, Don Lauro and Adrienne Chattoe-Brown, to conduct it. Carried out from March through June, 2012, this Evaluation Report presents the consultants’ major findings. 1.1.1 Summary of Findings As categorized in the Scope of Work (SOW) at Annex 1, the findings of this evaluation are summarized as follows: � Achievement of the RHSC within family planning and reproductive health (FP/RH) commodity security have at global level been generally positive and in some areas impressive; regional-level accomplishments are promising but more recent; and country- level impact has to date been fairly weak. (The Coordinated Assistance for Reproductive health supplies (CARhs), a group that works effectively “to address short-term supply crises that periodically befall countries” is the exception that proves the point. Other advances at global level, such as AccessRH, should eventually have significant impact on commodities at country level) � Structures and Processes that the RHSC has developed and supports are solid and reasonably productive; however, the composition and what has been accomplished through these structures are not fully or currently in line with the rapid membership growth that the Coalition has experienced. The strategic goals, principles and assumptions associated with the 2007-2015 Program Strategy need to be revisited to address current challenges, consider new opportunities, and move the Coalition forward. � Future Directions for the Coalition are to realign structure as well as strategy to reflect current and future membership; continue progress for commodity supplies at global and regional levels; and become more directed and committed to having impact at country level. 1. Executive Summary 3 Evaluation of the Reproductive Health Supplies Coalition 1.1.2 Evaluation Methodology Following the SOW, the methodology that the evaluators used included key informant interviews (in all, over 60 representatives of Coalition members and staff were interviewed) together with reviews of RHSC documents and other relevant literature (listed at Annex 2). In addition, as discussed with donors and included in the Evaluation Work Plan (Annex 3), the evaluators added an electronic survey of all members (65 members responded) and observed/participated in a few Coalition activities, such as Working Group conference calls and meetings. 1.1.3 Organization of the Evaluation and this Report With details covered in 19 key questions included in the SOW, this evaluation is organized around three main subject areas: Achievements; Structure/Processes; and Future Directions. As the RHSC donors agreed early in this process, within these three broad categories the evaluators reformulated the 19 questions into eight overarching questions. These overarching questions also provide the outline for the body of this Report (Sections 3 to 10), with the findings in relation to the 19 SOW questions provided in Appendix B. The remainder of this Executive Summary summarizes major findings within the three SOW core areas. 1.2 Achievements: Global, Regional, and Country Level The 2007-2015 Strategic Plan is a roadmap against which RHSC achievements may be assessed and measured. The RHSC Strategic Plan has three goals each encompassing three objectives, shown below as on the RHSC website: Goal One: Increase Resources Mobilize Additional Resources Enhance Use of Existing Resources Maximize Total Market Goal Two: Strengthen Systems Strengthening Functioning Systems Respond to System Failures Strengthen Response to Crises Goal Three: Add Value Foster Greater Harmonization Expand Knowledge Base Frame Global Efforts For the most part, Goals One and Two delineate what member organizations would accomplish on their own, collectively, and sometimes within Working Groups. However it is Goal Three that describes what the RHSC itself does: “Assure the added value of the Coalition as a productive and sustainable global partnership through efficiency, advocacy, and innovation”. Five years on, achievements of the RHSC and members have been considerable at the global level; somewhat productive and promising at the regional level, though recent and not necessarily replicable; and at country level both small scale and sporadic for the most part, particularly in relation to the many countries with high unmet need and corresponding weaknesses in supply systems to serve those needs. . In fairness, global efforts have been on RHSC’s agenda for well over five years while ability to reach out to countries is relatively recent. In addition, some global achievements have had impact in some countries. For example, through combined inputs of several donors and organizations, CARhs has alleviated stock-outs in several countries; the Pledge Guarantee for Health has effected price reduction for Implanon in Ethiopia; and advocacy for commodity security has made progress in countries such as Ghana and Uganda. 4 Evaluation of the Reproductive Health Supplies Coalition 1.2.1 Global The prominence of FP/RH commodity security has clearly increased since the 2001 Istanbul Conference first brought international attention to this issue. From 2005 onwards, including within the 2011 Access for All Conference in Addis Ababa, the RHSC has been at the forefront and increasingly the face of this movement. The Coalition has raised the profile of FP/RH commodity security on the global health agenda. Illustrative of advances are members’ engagement with the UN Commission on Life Saving Commodities for Women and Children to include specific contraceptives and maternal health supplies, and the HandtoHand (H2H) Campaign which has brought increased recognition to the importance of contraceptives as a strategy to reduce maternal deaths. H2H was also a forerunner to and underlays the rationale and impetus of the upcoming FP Summit. To promote global level advances, the Coalition has nurtured and its members have developed a number of tools that already have and will continue to increase and improve FP/RH commodity security for years to come. Among these are approaches that facilitate procuring large amounts of contraceptives more efficiently, resulting in cost and time savings, and sometimes lower product prices: � AccessRH to electronically receive and process orders and RHInterchange to track procurements are landmark developments widely used. AccessRH led to saving an average 10.3 weeks in wait-time in 2011 � The Coordinated Assistance for Reproductive Health Supplies Group (CARhs) reacts to and remedies stock-outs which threaten supply flows in 22 countries at risk. It also proved effective at addressing overstocks and resolving situations where stock levels are below minimum standards but not stocked out � Total Market Initiative, based on analysis of market segments for contraceptives, has been successfully applied if not fully realized in two countries, Honduras and Madagascar � FP/RH manufacturers appreciate the neutral space that the Secretariat occupies. This has led to better understanding of how market dynamics work within global supply chains and joint ownership of commodity security by procurers and suppliers. Better communication has resulted in significant price reductions in Implanon, Jadelle, and the female condom of at least $22m. � Advocacy by local civil society organizations of host country governments has advanced local capacities for contraceptive procurement and commodity security in several countries, most notably in Uganda and Ghana � The Pledge Guarantee for Health (PGH), although slower to take off than originally predicted, shows promise through its contributions to brokering the price reductions in the procurement of Implanon, which has to date been particularly beneficial for the substantial procurements made by the Government of Ethiopia � Product pre-qualification, moving forward steadily if somewhat slowly, is making both quality and availability the hallmarks for future procurements of FP/RH commodities The three RHSC Working Groups, and to a lesser extent the more focused Caucus on New and Underutilised Methods and Maternal Health Task Force, have been directly involved in and directly contributed to most of these global level accomplishments. It is important to sustain such global level efforts and in particular to be attentive to promoting and pursuing through-lines to country-level impact. In this regard it would be useful for the Secretariat and RHSC structures to encourage coordinated and consolidated action by members working at country level. For example, incentives could be built into the successor to the Innovation Fund for awards able to demonstrate collaboration and/or impact at country level. Similarly, the Secretariat could more thoroughly map the country presence of members, local as well as international organizations. Even established and largely successful global partnerships such as CARhs, which depend upon USAID, UNFPA, and other international donors for information 5 Evaluation of the Reproductive Health Supplies Coalition about supply shortfalls, could benefit from fuller knowledge about other member organizations working within particular countries. 1.2.2 Regional In recent years, two regional forums have joined the Coalition, the Latin America and Caribbean (LAC) Forum and the Sécurité Contraceptive en Afrique Francophone (SECONAF) for SubSaharan African French-Speaking Countries. For different reasons, each of these forums offers distinct advantages to their respective regions: � The LAC Forum provides a vehicle for the region to remain globally engaged with FP/RH as donor support for programs and commodities in the countries winds down; � SECONAF provides a new avenue for engaging with a large number of French-speaking African countries where FP/RH lags far behind. Regional memberships are recent within the Coalition and not necessarily a blueprint for reaching out to countries in the future. Though too soon to have demonstrated substantive achievements, progress made within these two forums, and particularly how advances are made, will be important in the coming years. The LAC Forum, for example, to serve needs within its region, is replicating the Working Group structure and the Innovation Fund mechanism of the RHSC. SECONAF in contrast is about to receive funding through a private foundation grant to RHSC to engage a regional consultant who will interface with countries in its region. Countries at different stages of development for FP/RH may well require different approaches. 1.2.3 Country The 2009 Terms of Reference states that “the RHSC commits itself to achieving a sustained supply of affordable, quality reproductive health supplies in low- and middle-income countries”. Having such impact does not mean that the RHSC operates fully within countries to advance FP/RH supplies and their availability. Rather it is the combined work of Coalition members that will ultimately produce progress at country level. The guiding principle that “increased country ownership is fundamental for reproductive health supplies” directs the RHSC toward involving and engaging countries in setting the agenda. This and other RHSC guiding principles derive from earlier work by the High Level Forum on best practices for global health partnerships. To date, progress at country level has been the least of RHSC accomplishments. The Resource Mobilisation and Awareness Working Group (RMAWG) has been the most directly involved at country level. This has largely occurred as a result of separately funded advocacy projects run by RMAWG members, and these results have been relatively small scale, though perhaps significant within a few countries such as Uganda and Ghana. The Market Development Approaches Working Group (MDAWG) also initiated in 2010 country level applications of the Total Market Initiative. However, selection of Honduras and Madagascar proved politically untimely, with results that emerged towards public sector understanding of private sector potential largely lost. More broadly, though more sporadically, the RHSC has engaged and made contributions within some countries. Its work in Ethiopia in particular, where price reductions for Implanon are resulting in millions in cost savings and increased access for hundreds of thousands of women, is a notable achievement. Instrumental in this occurring however was that the RHSC held its 2011 Annual Meeting in that country. Though annual meetings cannot be held to impact all countries, the principle and practice of price reductions in relation to volume may now be broadly replicated to have far-reaching impact. 6 Evaluation of the Reproductive Health Supplies Coalition 1.3 RHSC Structures and Processes The core structures of Executive Committee, Secretariat, and three Working Groups are all solid, even stolid, mechanisms through which the business of the RHSC is carried out. 1.3.1 Executive Committee The Executive Committee, particularly through leadership of a succession of strong chairpersons, has encouraged and presided over vast expansion of Coalition membership. The Executive Committee has also facilitated connecting RHSC with manufacturers for discussions about commodity security and supported including them as members of the RHSC. Most recently, the Executive Committee Chair provided impetus and inspiration for development and launch by the Coalition of the HandtoHand (H2H) Campaign. However, some seven years since initial formulation, the Executive Committee would benefit from being refreshed and reconfigured to more faithfully mirror the current membership of the RHSC. In addition, some greater depth of expertise and independent voices on the Executive Committee could increase capacity for providing long range oversight. For example, one or two Executive Committee seats could be set aside for notable experts, unaffiliated with member organizations, to further enrich discussions and perspectives on the RHSC. 1.3.2 Secretariat The Secretariat has faithfully and with great appreciation supported Coalition members. In the person of the RHSC Director, the Coalition has an effective facilitator, a nimble networker, and a leader who has worked well with the Executive Committee, including a succession of different chairpersons. He and the Secretariat have served the membership well by interfacing with those interested in FP/RH commodity security, including the growing number of new members. However, as RHSC membership has increased, new structures added, and the range of direct responsibilities increased, the Secretariat has become increasingly stretched. Resulting from some staff attrition as well as substantive expansion, some increase in technical expertise and experience at Secretariat level is indicated. Depending on new or expanded directions included within a revised and updated RHSC Program Strategy, additions to the Secretariat in terms of policy expertise, procurement proficiency, and country- level program experience should be considered. 1.3.3 Working Groups and other Coalition structures Working Groups and other RHSC structures, such as the Caucus for New and Underused Methods, the Task Force for Maternal Health Supplies, and the two regional forums, depend entirely on members’ voluntary participation, with ongoing and sometimes significant support provided by one of the Secretariat staff. The work that occurs within these structures embodies this “coalition of the willing” and exemplifies an organization that engages, as the Director states, in “the logic of leading from behind”. Most of the tools and approaches to advance FP/RH commodity security occur or have emerged from within these structures and this mode of operation. However, only relatively few of the many members are and can be actively engaged in these structures. Except within the forums, most new members have not in fact found ways to join with the long-term members who remain the stalwarts within these structures. That is a major challenge for the future – how to adapt current structures or re- structure so that there is wider opportunity for fuller engagement, especially by new members representing different constituencies. With such large increases in members having already occurred, the Working Groups and other structures have not correspondingly extended their reach. These RHSC structures thus risk becoming more an elite sub-culture of founding members than a vital and vibrant part of expanding interests and possibilities. 7 Evaluation of the Reproductive Health Supplies Coalition 1.3.4 Comparative advantage and added value The comparative advantage of the Coalition lies in its neutrality, its reach, and its resulting power as a convener. With growth, the RHSC has greatly expanded the diversity of its membership, horizontally (from governments to commercial firms) as well as vertically (from country-level NGOs to large international donors). The commitment of its members and its small, efficient Secretariat are also assets. Its limited spending power has also given it strength as this has helped keep the Coalition a neutral space. However with expansion the Coalition is outgrowing the principle of sole reliance on volunteerism. The 2007-2015 Strategic Plan had anticipated adding value in several areas. These include providing a forum for technical collaboration, bringing together a ‘brain trust’ of expertise and experience, and promoting effective and efficient use of limited resources. Members are also able to identify a broad range of global, regional and country level efforts to improve commodity security where the Coalition has added value. With specific reference to Goal Three, the Strategic Plan envisioned that participation in the Coalition would add value in eight ways. As summarized in the evaluators’ assessments against these expectations in Table 1 below these have been partially realized: Table 1 Expected added value of participation in the Coalition Expected added value Assessment Providing a forum for technical collaboration and problem-solving. Demonstrated from the outset. Increasingly being used as such by a wider variety of members. Bringing together a “brain trust” of expertise, experience, and learning relevant to finding solutions to RH supply security. Partially demonstrated. Wide and growing membership from different constituents, with effective communications between many of them. However country level members not sufficiently engaged as a resource. Promoting the efficient and effective use of existing, limited resources by improved coordination and harmonization of RH supply programs. Some progress e.g. through CARhs and RHSC engagement in a variety of initiatives, and coordination between partners improved. But greater harmonization not yet evident. Bringing about change through unity of purpose and action. Less emphasis on this in past, but growing. There may be potential for capitalizing on this more. Increasing global awareness of RH supply issues and strengthening advocacy and influence globally and in countries through shared objectives and messages. Demonstrated at global level but less evident at country level. Promoting innovation and collaboration among the public, private, and commercial sectors to develop markets for RH supplies. Not yet demonstrated, but new relationships have been formed which could lead to this. Initial work has been done on total market approaches. Members joining together to seek funds for innovative ideas, research, advocacy, and best practices. Some Innovation Fund awards demonstrate collaborative effort, but there is potential for more. Several RMAWG members are joining together to secure funds to continue advocacy at country and other levels. Maximizing the impact of investments in RH supply security by leveraging the investments of other participating organizations. Demonstrated. e.g. increased funding for RH Supplies from GFATM. Good leveraging from members. 1.3.5 Value for money It is difficult to assess value for money of the Coalition as a whole when none of its major initiatives (AccessRH, RHInterchange, CARhs, etc,) have themselves been completely evaluated. It is possible to identify however that the Secretariat is run efficiently and cost effectively, and as described above is essential to the operations of the Coalition. The Coalition has demonstrated good leveraging power to attract resources to RH commodity security initiatives. Including DFID’s recent support of nearly $16 million for CARhs, some $45 million in new funding has been leveraged since 2007. Coalition efforts have also led to 8 Evaluation of the Reproductive Health Supplies Coalition significant cost savings in the price of commodities of at least $22m. Contrasted with the $7.3m allocated by donors to the Secretariat for core costs and the Innovation Fund, generation of such savings alone would seem to be a particularly good return on their investment. 1.3.6 Strategic plan and Monitoring and Evaluation Framework The 2007-2015 Strategic Plan has generally served the Coalition well. However, the expanded membership, current conditions, and future possibilities suggest that it should be revisited in the near future. The goals are still highly relevant for the Coalition but a major revision of the monitoring and evaluation framework is needed. As it currently stands it is too wide, does not fully capture what the Coalition has or could hope to achieve, and is not used as a management tool or to promote accountability. Development of an overt theory of change, currently absent from though implied within the strategy, would facilitate development of a more useful and relevant Monitoring and Evaluation Framework (MEF). 1.4 Future Directions and Recommendations for the Coalition With membership likely to continue increasing, the future of the RHSC depends on three interrelated factors: � Continue member commitment, widen involvement to newer constituencies, and sustain the sense of ownership that have been core to success within the RHSC � Renew and moderately increase funding to support the RHSC, particularly the value- added contributions of the Secretariat, including modest increases in staffing and a reoriented re-granting mechanism (analogous to the Innovation Fund, but with modifications) to support activities and projects generated by members � Refocus and revise the Strategic Plan to direct the Coalition towards having impact further along the supply chain, notably all the way to country-level to reach and ultimately serve more who wish to use RH commodities . To support movement towards the above, the evaluation team recommends the following principles and practices for the RHSC: � Retain the core focus on RH commodities � Maintain global as well as local focus on supply chain improvement � Promote greater ownership of the Coalition by country-partners: government, civil society, and other country-level organizations � Retain but supplement the partner-centric funding model by renewing, renaming, and reorienting the Innovation Fund and supplementing operations of RHSC structures � Revise and update the Strategic Plan and support its implementation by adding staff with requisite experience and expertise to the Secretariat � Continue selective proactive leadership, including taking clear public positions to move the FP/RH agenda forward The Coalition builds on a productive past derived from a straightforward central aim: move FP/RH commodities and supply chains, often afterthoughts, to the forefront. Though awareness and prominence about the importance of FP/RH supplies and products has increased since the Coalition has been in place, the RHSC should keep its core focus on commodity security. From this centre, however, the Coalition can do more to ensure that country members, especially those in countries where serious supply problems persist, are fully involved in finding solutions. As the RHSC maintains global and local focus on supply chains, reliance on partners and their voluntary contributions may also need greater supplementation and support. To this end, the evaluators suggest that the RHSC would put additional funding to good use. This should include not only some staff additions at 9 Evaluation of the Reproductive Health Supplies Coalition Secretariat but also renewed (and increased) support for a new Fund to support Working Group generated and country-inspired initiatives. The latter should provide some needed adjustments to support greater Global South participation and catalyze problem-directed solutions further along the supply chain. The current partnership with the Global Programme of the UNFPA as well as future arrangements to come out of the Family Planning Summit may both be able to make this happen. With membership having grown rapidly over the last few years and likely to continue doing so in the future, the RHSC is positioned and poised to continue contributing to RH commodity security. As in the past, it will do this by both adding to this field of knowledge and applying practical tools and approaches to improve systems at various levels. However, the Secretariat in particular does not currently have the bandwidth or depth among staff to do so. Furthermore, the RHSC Strategic Plan is somewhat outmoded and out-of-date. Unlike its broadened membership base, the current Strategic Plan does not contain cohesive pathways towards moving effectively and more fully along the supply chain. As a membership organization, the Coalition needs to identify means to involve members within this fundamental reformulation. It is neither too late nor too soon to begin doing so at the next Annual Meeting. An immediate challenge for the Coalition is to secure future funding even as it recasts itself for more involvement of members and moves toward fuller country level engagement. The evaluators have found sufficient productivity and promise within the RHSC to support application for continued funding. Leadership within RHSC remains strong, the Secretariat solid and well appreciated, and members increasingly well represent FP/RH and the collective needed to move forward. Structures and operations to date provide a solid base of experience upon which to improve. The future – needs as well as opportunities – make the Coalition a best case investment for going the next and last miles to make FP/RH available for all who wish to use them. 10 Evaluation of the Reproductive Health Supplies Coalition Among human endeavours, our highest aspirations and greatest potential are in working together towards a common goal. Collaboration among member organizations to ensure contraceptive and reproductive health commodity security is the founding principle of the Reproductive Health Supplies Coalition (RHSC or the Coalition). This external evaluation focuses on how well and with what results the RHSC has operated. As articulated in its 2007-2015 Strategic Plan, the RHSC vision is that “All people in (low and middle-income) LMI countries are able to access and use affordable, high-quality supplies, including a broad choice of contraceptives, to ensure their better reproductive health”. As specified in its 2009 Terms of Reference, “achieving (this) vision requires a multi-stakeholder approach to improve RH behaviours. RHSC’s membership includes multilateral organizations, bilateral and private foundation donors, low- and moderate-income country governments, civil society, intergovernmental organizations and NGOs, and the private sector”. The 2007-2015 Strategic Plan of the RHSC articulates three strategic goals: 1. Increase the availability, predictability, and sustainability of financing for RH supplies 2. Strengthen the capacity of health systems to deliver RH supplies in a sustainable manner 3. Assure the added value of the Coalition as a productive and sustainable global partnership through support for efficiency, advocacy, and innovation 2.1 Scope of Work The purpose of this external evaluation is to assess what the Coalition has achieved more than midway through its Strategic Plan. To determine the effectiveness, efficiency, results, and impact of the Coalition and its partnership with nearly 200 member organizations, this evaluation addressed three core objectives from the Scope of Work (SOW): � Achievement: Assess how well has the RHSC met its objectives and made an impact on Reproductive Health commodity security � Structures/Processes: Assess how well have RHSC structures and processes worked towards achievement of RHSC objectives � Future Directions: Recommend how should the RHSC's role evolve in light of the current and emerging environment, including measures to improve the efficiency and effectiveness of the RHSC To guide this evaluation, the SOW also included an extensive set of 19 additional questions (and sub-questions) aligned with these three evaluation objectives. (The complete SOW is at Annex 1.) During a mid-March planning meeting, the external evaluators and the donors supporting RHSC reviewed, discussed, and prioritized these SOW questions. Based on this, the evaluators subsequently summarized and included eight overarching questions in the Evaluation Work Plan (in Annex 3). 2.2 Organization of the Report The body of this report is organized around the eight over-arching questions. Evaluator findings in relation to the subset of the 19 key questions in the SOW are in Annex 4. The Executive Summary recapitulates overall findings in relation to the three objectives of this evaluation: Achievement; Structure/Processes; and Future Directions. 2. Introduction 11 Evaluation of the Reproductive Health Supplies Coalition 2.3 Methodology Staff from the four major donors providing core costs for the RHSC and its Secretariat developed the SOW for this evaluation. This included a methodology using two principal approaches: a review of relevant literature and in-depth interviews with key informants. During a planning meeting between these donors and the evaluators, on March 15 in London, an electronic survey of RHSC members was added (Annex 5). Also, as discussed in this meeting and included in their March 20th work plan, the evaluators added a complementary approach: to a limited extent and within the evaluation timeframe, participation in and observation of RHSC operations and events. This four-part methodology and analytic approaches are detailed in Annex 6. 12 Evaluation of the Reproductive Health Supplies Coalition 3.1 Overview The RHSC has had demonstrable impact on commodity security at global level, made some promising progress regionally, and to date had some, but relatively less, impact at country level. At global level the tools nurtured by the Coalition, the Pledge Guarantee for Health (PGH), AccessRH, the RHInterchange and the Coordinated Assistance for Reproductive Health Supplies Group (CARhs), have demonstrably improved coordination, availability of information, prices for commodities, and effective use of donor money. The Coalition has also influenced the global health agenda through the HandtoHand (H2H) Campaign by increasing recognition of the importance of contraceptives as a strategy to reduce maternal deaths. The Coalition has contributed to ensuring that family planning commodities are included in the UN Commission on Life Saving Commodities for Women and Children. The Coalition has also improved commodity security by facilitating coordination between the main donors engaged in commodity security. The RHSC provides a forum for a wide range of stakeholders to come work together on commodity security, an essential role in such a diversely populated and complex field. It has in addition encouraged the participation of manufacturers who have traditionally been ‘outside the fold’ of commodity security partners. (This is discussed further under Section 4.1 on Comparative Advantage and Section 4.2 on Added Value.) At regional level progress has been made by encompassing the LAC forum within the Coalition and supporting its application of RHSC structures. However, the Francophone forum, SECONAF, has only recently joined and is too new to have shown results. CARhs and the PGH have directly improved the availability of commodities at country level. AccessRH and the RHInterchange are also likely to be improving availability, although this is difficult to prove. The HandtoHand Campaign has probably not improved country level commodity security directly – it is too new and its through-line to countries is not clear. While engagement by people from developing countries in the work of the RHSC has been limited, the Coalition has opened the door to dialogue by securing the membership of representatives of a number of governments: China, Ethiopia, Ghana, Mauritania, Nicaragua, Nigeria, Senegal, Tanzania, Uganda and Zambia. Additional support for the work of the Coalition via Project RMA and Advance Family Planning has encouraged advocacy of governments by civil society and built some local capacity in advocacy for commodity security. The contribution of the Coalition to these efforts has been through a variety of means. These include generating ideas, widening engagement of stakeholders in initiatives, supplying seed- corn funding, facilitating implementation, encouraging technical rigor, and encouraging the use of Coalition-associated tools. 3. What contributions has RHSC made and what impact has it had on improving commodity security at global, regional, and country levels? 13 Evaluation of the Reproductive Health Supplies Coalition 3.2 Performance against the monitoring and evaluation framework This section reviews the performance of the RHSC as measured by the monitoring and evaluation framework (MEF). Section 5 discusses the strengths and weaknesses of the MEF. A summary table attempts to systematically assess how well the RHSC has performed (see the table in Annex 7 and its far right column, ‘progress’) but the absence of targets makes this subjective. The MEF does not currently capture indicators for performance at regional level. (The most recent monitoring report provided by the Secretariat is in Annex 13.) 3.2.1 Impact at global level At global level there has been an increase in donor funding for contraceptives worldwide and an increase in the level of funding allocated to RH supplies by The Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM). However, contributions from both have decreased recently largely as a result of the global economic turndown including the suspension of the GFATM’s 2011 funding round. However four new donors have entered the commodity security scene (as measured by new donors to the Global Programme to Enhance Reproductive Health Commodity Security (GPRHCS)). Volatility of donor funding is being addressed by the Pledge Guarantee for Health, which has so far channelled $12m for implants in Ethiopia. AccessRH is now established within UNFPA and has dealt with $32m worth of orders. Despite current constraints – only condoms on stock and full payment required in advance for orders – AccessRH has already had orders totalling $32m. At the same time, the pre-qualification initiative is endeavouring to improve quality of available products and has pre-qualified 33 more products since 2007. These include seven contraceptives of which only two are lower-priced generic products. Much accomplished, much more remains to be done. 3.2.2 Impact at country level The report against the MEF shows that all focus countries1 now have commodity security coordination committees established; more countries are accessing GFATM funds to support RH commodities; expenditure of national budgets for contraceptives has improved in many places, with many countries spending 100% or more of their allocated budgets; and some countries are starting to be aware of or use the tools available to them e.g., AccessRH, PGH, and RHI. Donor engagement at country level through the CARhs group has successfully averted stock-outs. Nevertheless, stock-outs across the 14 focus countries still show very mixed results: some countries are getting worse; substantive changes in commodity security through adopting a total market approach is not yet evident; and only one country of the four currently measurable shows a growth in the non-public sector share for key contraceptive commodities. 3.3 Country uptake and results of RHSC-inspired initiatives The RHSC has engaged with commodity security initiatives in a variety of different ways. Some it has funded directly, for example through the Innovation Fund. Some the RHSC has facilitated through opportunities for networking and technical exchange among partners. The RHSC contribution to these initiatives is discussed below in section 3.6. The evaluation team assesses the following initiatives as those most closely associated with the RHSC and the most significant for countries, either in terms of proven or potential impact: _________________________ 1 14 focus countries (Burkina Faso, Ethiopia, Ghana, Mozambique, Rwanda, Tanzania, Uganda, Guatemala, Honduras, Mexico, Nicaragua, Bangladesh, Mongolia, and Nepal) were agreed by the RHSC on the basis that they are or were receiving targets support by RHI, Project RMA, and / or UNFPA’s GPRHCS. 14 Evaluation of the Reproductive Health Supplies Coalition 3.3.1 Coordinated Assistance for Reproductive Health Supplies Group The Coordinated Assistance for Reproductive Health Supplies (CARhs, formerly “Countries- At-Risk”) Group was established in December 2004. The CARhs brings together key commodity suppliers to address short-term periodic supply crises at country level. The group currently reviews data on 22 countries monthly. The table in Annex 8 shows the outcome of CARhs engagement from Oct 2010 – Sept 2011 The evidence suggests that CARhs is most effective at addressing overstocks (by delaying or cancelling future shipments), but is also effective at resolving situations where stocks are below minimum levels but not stocked out. The 2012 monitoring report (see Annex 13) reports that CARhs “saved money by arranging the transfer of more than US$2 million in excess commodities from Rwanda to Burkina Faso. It played a pivotal role in clearing millions of dollars’ worth of RH supplies stuck in port in Nigeria. In addition, it assisted with the procurement of six million condoms for a new military HIV/AIDS prevention program in Burundi.2” From the outset CARhs has impacted positively on commodity security at country level. Donors and implementation partners have worked together more closely with a systematic approach to address problems in which they all have a stake. Country participation has however been limited. Some stakeholders maintain that civil society and other country-based members could be part of an early alert system for potential stockouts. This deeper systemic approach to addressing stockouts has not yet been taken forward. 3.3.2 Pledge Guarantee for Health The Pledge Guarantee for Health (PGH) is a financial tool that enables grant recipients (i.e. countries) to obtain short-term, low-cost commercial credit on the basis of pending aid commitments. This enables countries to buy donor funded commodities as needed rather than waiting for donor commitments to be realised. PGH was launched in October 2009 with the intention of improving the predictability and effectiveness of donor aid for RH commodity procurement. It is also now marketed as potentially able to reduce prices of commodities when used for large scale procurements, thereby enabling committed resources to go further. PGH operates within the United Nations Foundation. The PGH has taken longer to get underway than many had originally predicted. . High numbers of transactions were expected within a short timeframe. However, PGH has yielded only two successful executions so far. One of these was to support malaria bed nets in Zambia. However in the other case, the PGH contributed to significant price reductions in the sale by Merck of $12m worth of Implanon contraceptive implants to Ethiopia, reducing the cost per unit from $18 to $16.5. Lacking ongoing support for operating costs, the future of PGH is highly uncertain. This is unfortunate as the PGH has the potential to significantly impact commodity security at country level. 3.3.3 AccessRH AccessRH was launched in 2010 at UNFPA with design and financing support from multiple Coalition members. The purpose is to reduce volatility in supply resources though high- volume procurement, warehousing of commodities for rapid delivery, and transparent information on pricing and product costs. _________________________ 2 RHSC Monitoring Report (2009-2011), p23 15 Evaluation of the Reproductive Health Supplies Coalition Between Jan 1st 2011 and Mar 31st 2012 the value of condom orders (the only commodity currently held in stock by AccessRH) has exceeded $32m. Of this, $4.5m has been from UNFPA country offices (i.e. the captive market for the service which could be serving either governments or local organisations) and the remaining $27.5m has come from third party clients3. In comparison to previous years before AccessRH was set up, the number of external clients in 2011 has increased, with a larger proportion of these being new clients (74%). This implies a country level demand for this service and growing country reach. Shipping condoms from stock rather than production has led to a 10.3 week wait time saving in the course of 20114. Further improvement may occur when the plan to share forecasts for other commodities with manufacturers comes online to help them deliver on schedule and improve estimations of shipping dates. 3.3.4 Reproductive Health Interchange The Reproductive Health Interchange (RHI) provides open access to up-to-date, harmonized data on approximately $1 billion worth of shipments of contraceptive supplies for over 140 countries. In 2011 RHI migrated from John Snow Inc. (JSI) to UNFPA becoming part of AccessRH. The intention is to improve the quality and availability of data needed for planning, ordering and managing procurement. At the end of 2011 there were 632 active users registered, a substantial increase from 304 in 2009. Data from RHI is fed into the CARhs process and all of the Coalition’s 14 focus countries are registered users (which enables them to use the system for tracking shipment data). A total of 92 different countries have registered as users. User logins, an indication of the use of detailed reports, have ranged from about 22 to 35 per week during 2011, an increase on previous years implying regular use of the RHI. Ethiopia, Pakistan, Uganda, India, Kenya and Zambia all appear to be frequent users. 3.3.5 Hand to Hand campaign The Hand to Hand Campaign was launched in 2010 with the intention of mobilising partners to achieve 100 million new modern contraceptive users in the next five years. Members are encouraged to support the campaign by increasing resources and service provision, making more contraceptives available at affordable prices, improving systems and carrying out international and national advocacy efforts. The Campaign has been unusual for the Coalition in that it relies primarily on the efforts of the Secretariat for implementation rather than partners. The Campaign, if successful, could have a major impact on countries. It was not possible within the scope of this evaluation however to assess the extent to which commitments to the Campaign represent additional efforts by the partners or whether the Campaign is just re-framing or perhaps refocusing existing efforts 3.3.6 Initiatives funded through the innovation fund Some of the initiatives supported by the Innovation Fund have been intended to have some impact at country level. The table in Annex 9 summarizes results for those initiatives that have been completed. _________________________ 3 In 2011 these external clients consisted of 28 Ministries of Health or other public sector agencies (e.g. Social Security Institute, Ministry of Population etc.), 10 NGOs, 13 UNDP offices and 3 other UN Agency offices. 4 Source: AccessRH slides from UNFPA 16 Evaluation of the Reproductive Health Supplies Coalition 3.4 Impact at regional level The MEF does not capture results at regional level. However the RHSC Director reported that the LAC Forum has built a community of interest around commodity security; brought the Inter-American Development Bank in as a new donor in this area, as well as attracted funds from UNFPA and USAID; brought Latin America into the wider RHCS community; and created a link between the global community and countries in the region (viz. it helped solve a problem between a global manufacturer and a service provider NGO in the region). The francophone forum SECONAF is too new to have had an impact on regional commodity security. However, initial work supported by an Innovation Fund grant to Equilibres & Populations, has sparked interest among a range of stakeholders, notably including the Hewlett Foundation and the French government, and encouraged some initial advocacy efforts by civil society. 3.5 Influence on the global health agenda The structure and operations of the RHSC are such that it is difficult to tease out the exact contribution of the Coalition as a whole, rather than the influence of individual members. However there are examples where there has been demonstrable added value of the Coalition to the global health agenda. Members of the RHSC working together helped to get family planning commodities included in the UN Commission on Life Saving commodities for Women and Children. The RHSC then contributed directly to the work of the Commission by advising on the selection of commodities to be included and carrying out data collection and analysis. The RHSC has also helped to bring provision of commodities into the maternal health debate by initiating the Hand to Hand Campaign. Part of the rationale for its development was to rally the support of the family planning community around the maternal health agenda, in particular around “Every Women/Every Child”. The H2H Campaign can be seen as the forerunner of the July 2012 Family Planning Summit which represents a widening realization of the importance of contraceptives to achieving MDG 5 and indirectly MDG 4. Generally the RHSC is helping to move family planning into a more central position; previously family planning had been side-lined by or subsumed into wider agendas, such as maternal health and HIV/AIDS, and had consequently lost focus. RHSC has also helped reduce the distance between the family planning and maternal health communities. The RHSC has also helped to demonstrate the potential contribution of manufacturers to the wider development agenda, positioning them as partners in family planning rather than adversaries, as has been the case in some global health initiatives. This has relevance to the work of UNITAID, the Clinton Health Access initiative, and bilateral donors working to improve access to medicines. From a regional perspective, feedback from interviewees in the LAC forum suggests that engagement of the Coalition in this region has helped to retain some global level attention on a region which is no longer a recipient of significant amounts of aid monies, but which still has significant commodity security issues for a large population. 3.6 Contribution of the Coalition The contribution of the Coalition to these efforts has been through a variety of means. These include: 17 Evaluation of the Reproductive Health Supplies Coalition � Generating ideas – by members of the Executive Committee and the Secretariat, by partners in Working Groups, or by members attending other Coalition meetings � Widening engagement of stakeholders in initiatives – serving as a forum for wider member engagement on Coalition-inspired initiatives or those carried out by individual partners � Providing funding – through the Innovation Fund � Facilitating implementation – hosting discussions, sitting on advisory boards � Encouraging technical rigor – reviewing progress, ensuring appropriate stakeholders are engaged, providing institutional memory on initiatives � Advocating for the use of Coalition-associated tools – through the website, annual meetings, strategic communications encouraging advocates in the Coalition to highlight the work of the RHSC A summary of Coalition contributions to key initiatives is at Annex 10. 18 Evaluation of the Reproductive Health Supplies Coalition 4.1 Comparative advantage 4.1.1 Neutrality The main comparative advantage reported to the team was the Coalition’s neutrality: space to meet for jointly and transparently addressing common concerns. Many of the members have had a long history of ‘issues’ with each other and some have been strongly territorial; in the early days before 2009 some used the Coalition as a space to revisit these. Many of the members were also in natural competition with each other, e.g. by being potential recipients of donor funds, or in search of outcomes they could claim. However, before and since 2009 successive Chairs and the Director have pro-actively and constructively worked to alleviate this and build more trust. Even though some of those issues and tensions still exist, suppliers are now able to engage strategically with procurers for example. Trust has also created a more constructive relationship with UNFPA regarding its Global Programme on RH Commodity Security. Operations of the Coalition, leading to some added value for members (as discussed in section 4.2), also have overcome or at least neutralized competition among members. The location of the Secretariat outside the US or the UK has also contributed to a sense of neutrality. This neutrality has been essential to promoting joint accountability, emphasizing the collective interest in making initiatives work and commodity security better. Neutrality is a great asset to the Coalition and its preservation is important. 4.1.2 Reach across field (breadth of membership) The Coalition has an unusually wide range of constituents – many of the key global players in RH commodity security are represented and some of the key actors at country level. The presence of the manufactures is particularly significant: prior to this they were not engaged with the wider RH supplies community as a group or with some individual members other than through commercial relationships. Their inclusion was strategic and has facilitated considerable gains in commodity security. For example, the Implanon Initiative and the other price improvements is based on better understanding of what manufacturers need to enable more cost-effective production. Other examples of the benefits of membership breadth include stakeholder engagement around pre-qualification and the range of signatories to the HandtoHand Campaign. 4.1.3 Reach up and down field (from countries up to global level donors) The Coalition is also notable in its depth of membership from small country-level NGOs to national governments to international donors. Clearly there are issues with how active the government and NGO members are and can be in the Coalition’s present formation. Nevertheless, the opportunity for different constituents to access one another at annual 4. Over and above partner organizations’ own accomplishments, what is RHSC’s comparative advantage, added-value, and value-for-money? 19 Evaluation of the Reproductive Health Supplies Coalition meetings is regarded as a benefit certainly by some of the small NGOs (in particular during the annual meetings in Kampala and Addis Ababa). There is potential for making more of this benefit. 4.1.4 Power as a convenor This breadth and depth of membership and the Coalition’s neutrality gives it a large and powerful network on which it can draw. According to one interviewee this is slowly being translated into “a community beginning to speak the same language”. This gives the Coalition and therefore its cause, greater weight in the international development arena. Although it has been used on occasion (e.g. to ensure contraceptives were in the UN Commission on Life Saving Commodities for Women and Children), this may be an under exploited advantage. For example, reaching common positions, as was done on the use of pre- qualified commodities, may also be applied for positioning the Coalition as a greater force in international development. The evaluation team recommends that the Coalition consider how it can gain greater leverage from its power as a convenor and use it for advocacy. Convening power also translates into a valuable resource for many members. The annual meetings, though time intensive to organize and attend, are valued by many participants because of the access they gain to other members from different constituencies. As one interviewee said “You’ve got all the players in reproductive health in one place for two or three days – that’s enormously powerful”. Several respondents to the member survey similarly remarked on the value of connections made at the annual as well as regional meetings. The frequent exchange of information through the newsletter, special announcements, and the website are also generally well regarded. The role of the Secretariat in the middle of this gives the RHSC staff, especially the Director, enormous scope, on which they capitalize, to bring people into discussions, raise issues, and convene meetings and discussions in various forms. This is also enabled by the Coalition structure which is fairly flexible and allowed for the opportunistic inclusion of LAC, the development of the cross-cutting Caucus on New and Underutilized Methods and the new Task Force on Maternal Health Supplies. There are issues with how well these modalities work to include southern partners, as discussed later, but at least for the northern partners they work well. 4.1.5 Commitment of members The commitment of members was notable among those we interviewed and is evinced by the time that some organizations spend in Coalition activities without immediate or direct return. One senior figure at an international NGO anticipates spending between half to one day a week on his new role as a Working Group head (and believes it would merit two days if his available time was not a restriction). The International NGOs and the technical partners are particularly aware of the costs and benefits to them of membership and clearly make informed and considered decisions about the value of engagement. It will be important to the future of the Coalition that time commitments do not become too onerous. The investment by the bi-lateral agencies is also notable. USAID, for example, had the team interview seven staff who either were or had been very engaged with the work of the Coalition. Even DFID, which is structured very differently with a much smaller technical staff, has two actively engaged staff members, one of whom is Chair of the Executive Committee. 20 Evaluation of the Reproductive Health Supplies Coalition 4.1.6 Committed, effective and small Secretariat Feedback on the Secretariat and its staff was almost entirely positive. The team evidently works extremely hard and this is recognized and appreciated by members. In most cases they appear to have had the right touch in terms of adopting a largely neutral but constructive position, facilitating the Working Groups, dealing with controversial issues, and managing the politics which were especially difficult in the early days. There is a lack of unanimity over the extent to which the Director should adopt a leadership function rather than just a convening role, but in the majority of instances he and his team seem to hit the right balance. The Secretariat’s limited funding, too small to fully serve current needs, has meant that activities have to be prioritized carefully. These issues are further discussed in section 7. PATH has also managed to maintain an appropriate relationship with the Secretariat and the wider Coalition. Competitors, alert to potential commercial disadvantage, do not seem overly antagonistic towards PATH and do not seem to bear ill will towards the Secretariat because of it. This has helped with the perceived neutrality of the Coalition. 4.1.7 Limited spending power The Coalition was funded on principles of volunteerism. While this has been a constraint to participation by some members (further discussed in section 7), it has also reduced the level of competition for access to resources because there have been very few Coalition controlled resources available (essentially only the Innovation Fund). It has also encouraged partners to come forward with funding and projects that can support the work of the Coalition, and has helped keep the Coalition as a neutral space. The vast majority of our interviewees had positive things to say about their experiences with the Coalition. Noticeable were criticisms around access to Innovation Fund grants if they failed to get one. The Coalition has been aware of this and is using the remaining Innovation Funds to support activities by country level organizations. It is wary of the perils of having substantial funds available in a competitive field. Further funding for specific things to be controlled by the Secretariat on behalf of the Coalition may be valuable for the future, as discussed later. However, this should be very carefully managed so that it does not upset neutrality, and thereby increase competition within the Coalition. 4.2 Added value The Strategic Plan had envisioned that participation in the Coalition would add value in eight different ways. Table 1 gives the evaluators’ assessment of the realization of this added value. 21 Evaluation of the Reproductive Health Supplies Coalition Table 1 Expected added value of participation in the Coalition Expected added value Assessment Providing a forum for technical collaboration and problem-solving. Demonstrated from the outset. Increasingly being used as such by a wider variety of members. Bringing together a “brain trust” of expertise, experience, and learning relevant to finding solutions to RH supply security. Partially demonstrated. Wide and growing membership from different constituents, with effective communications between many of them. However country level members not sufficiently engaged as a resource. Promoting the efficient and effective use of existing, limited resources by improved coordination and harmonization of RH supply programs. Some progress e.g. through CARhs and RHSC engagement in a variety of initiatives, and coordination between partners improved. But greater harmonization not yet evident. Bringing about change through unity of purpose and action. Less emphasis on this in past, but growing. There may be potential for capitalizing on this more. Increasing global awareness of RH supply issues and strengthening advocacy and influence globally and in countries through shared objectives and messages. Demonstrated at global level but less evident at country level. Promoting innovation and collaboration among the public, private, and commercial sectors to develop markets for RH supplies. Not yet demonstrated, but new relationships have been formed which could lead to this. Initial work has been done on total market approaches. Members joining together to seek funds for innovative ideas, research, advocacy, and best practices. Some Innovation Fund awards demonstrate collaborative effort, but there is potential for more. Several RMAWG members are joining together to secure funds to continue advocacy at country and other levels. Maximizing the impact of investments in RH supply security by leveraging the investments of other participating organizations. Demonstrated. e.g. increased funding for RH Supplies from GFATM. Good leveraging from members. This overall added value, and the comparative advantage of the Coalition as a whole, plays out differently for individuals, member organizations, and their combined efforts to address RH commodity security at country, regional and global levels. Added value for individual members and for country level initiatives primarily resides in the access provided to other members and information. According to the members who have worked at regional level, a further major added value to the Coalition has been in the support it has given to overcoming language barriers. (These are further detailed in Annex 11, Added Value of the Supplies Coalition. Comments specifically by southern members are page 24 of Annex 5, Coalition Member Survey 2012 Results Report.) The Coalition has added value to the work of individual partners working at global level by: � Changing the landscape in which they work, from wary and sometimes openly hostile to one in which they can work more effectively together. This has provided a space where competitors can collaborate. � Bringing down the price of commodities. In addition to the general benefits of this to donors, recipients and manufacturers (because it came with increased orders), it has helped at least one of the bilateral donors deliver on its commitment to reduce the price of RH supplies � Contributing to a harmonized approach for quality assurance of health products and how to move forward. � Ensuring common ownership of initiatives which might not have been as well received had they been perceived as being led by one donor or technical agency. The reach of tools and initiatives is extended when they are owned by the Coalition. � Providing easy access to stakeholders who can be consulted about the development of a tool and support its dissemination. RHInterchange was reported to have developed in this way. The Coalition has also provided continuity in the difficult process of developing new 22 Evaluation of the Reproductive Health Supplies Coalition tools. As one interviewee said “The transition from spawning an idea to implementing it can be a fraught one in terms of how you keep it true to its original impact intent and make it practical organisationally for it to run and operate. The Coalition’s value added is providing that continuity point, so that there can be check in, corrections made, or changes to expectations. That’s really helpful.” This was beneficial for the development of AccessRH. � Being a source of intellectual capital on RH commodity security, bringing all the key players and ideas to one place. � Acting as a central connecting point between different networks. � Providing technical agencies which are usually supported by only one donor, with alliances that reach beyond their traditional ‘camp’. A possible lesson learned is for the RHSC to stay engaged with some initiatives even after they have spun off to other organizations. The neutrality and cohesion of the Coalition could perhaps help to maintain momentum for such initiatives as AccessRH and PGH. In addition, ongoing activities, such as country-level advocacy under the RMAWG, may be linked with longstanding efforts, such as CARhs work to alleviate stock-outs. 4.3 Value for money 4.3.1 Overview The table below provides a comprehensive compilation of funding for and related to the RHSC. It does not include for example, the large amount of staff time that is provided by partners to enable the Working Groups to function, or attendance at annual meetings. This cannot easily be quantified and it was outside the scope of our evaluation to try to do so. Table 2 Expenditure on Coalition related activities USD 2009 2010 2011 Donors support to Secretariat 1,304,000 1,897,000 2,086,000 Innovation fund (awards) 712,404 289,419 631,465 Funding for RHSC-related initiatives, e.g. CARhs 7,923,000 10,665,000 23,523,243 Total expenditure on RHSC related activities 9,941,393 12,852,609 26,241,539 It is very difficult to assess whether this overall funding is value for money. The value for money of RHSC-related initiatives spun off from activities started within the Coalition has not been calculated and it is beyond the scope of this evaluation to attempt to do so. It would however be worth considering when detailed assessments of initiatives such as CARhs and AccessRH are carried out. In the absence of this information, there are other ways to look at the issue: 4.3.2 Cost effectiveness of the Secretariat Between 2009 and the end of 2012 the four donors to the RHSC Secretariat will have contributed $7.3m. Table 3 shows RHSC Secretariat expenditure for the period 2009 – 2011 with an estimate for 2012. Expenditure shown covers both the dedicated Secretariat team and supporting activities by PATH. For example the line for personnel costs covers Secretariat staffing in the Brussels office, Washington DC, and staff costs charged by PATH for regular and occasional support to the Secretariat. e.g. preparation of donor reports. The other large line item, operations, covers telephone, printing, office costs, insurance, facilities, etc. All other lines are fairly minimal. 23 Evaluation of the Reproductive Health Supplies Coalition Table 3 RHSC Secretariat Income and expenditure 2009 – 2012 USD Yr 2009 (actual)` Yr 2010 (actual) Yr 2011** (actual) Yr 2012*** (planned) Total 2009 - 12 % Expenditure Personnel 966,000 1,185,000 1,412,000 1,455,000 5,018,000 68% Consultants 7,500 106,000 115,000 20,000 248,500 3% Secretariat Travel 82,000 155,000 154,000 115,000 506,000 7% Subcontracts 0 129,000 61,000 0 190,000 3% Annual Meetings 45,000 41,000 24,000 65,000 175,000 2% Participant Travel 16,500 49,000 34,000 55,000 154,500 2% Operations 187,000 232,000 286,000 338,000 1,043,000 14% Total Expenditure 1,304,000 1,897,000 2,086,000 2,048,000 7,335,000 100% Notes on the table: *Plus interest **Additional funds received/spent on annual meeting from BMGF ***Plus approximately $150k in other funds towards forums and working groups Numbers have been rounded and do not exactly match donor financial statements. The phrase frequently used by key informants about the Secretariat was ‘lean and mean’. It has a small staff whose efficiency and effectiveness was praised by a variety of stakeholders. It is generally recognised that they are working beyond capacity. The proportion spent on staff (68%) would seem to be in line with the nature of the Secretariat’s work – largely networking, convening and communicating. The arrangement with PATH enables some specialist functions (e.g. budget administration) to be bought in regularly or occasionally on a fractional basis. Other specialist services such as HR are also available, covered by the PATH overhead. Outsourcing is used appropriately e.g. for website maintenance. Consultants are used selectively for largely strategic pieces of work and this is appropriate. Input from PATH enables the Secretariat to make best use of the donor funds at its disposal. The four donors have different rules about overhead, procurement, staff costs, etc. and the funds have flowed at varying intervals over different periods. The project administrator is skilled in matching the needs of the Secretariat with the funds available , and relieving Secretariat staff of this detailed process. In addition to donor funding, the Secretariat has sometimes mobilized contributions from partners to support ad hoc activities. These are usually in the form of support for travel to events, meetings. etc. Often these are provided directly to simplify processing. The Secretariat is essential to the running of the Coalition. It is the ‘glue’ that keeps it all together and enables partners to be effective in the Coalition ‘arena’. The quality of its work is reported by key informants to be good. Its value for money would also seem to be fairly high. 4.3.3 Value for money of the Innovation Fund The grants through the Innovation Fund have been fairly small (the largest was less than 200k). (A summary assessment of completed activities with results at country level is at Annex 9.) On the whole those activities seem to have been worth doing. The applications to the Fund were carefully considered, guidance on selection has been followed, and weak applications were rejected. For those applicants who had the capacity to formulate quality 24 Evaluation of the Reproductive Health Supplies Coalition proposals, the application and selection process seems appropriate for the scale of the grants and what they were trying to achieve. Members participating in the internal review suggested a number of improvements in operating the Innovation Fund or something similar going forward. (This is discussed in Section 7.4.2). 4.3.4 Leveraging power of the Coalition Reference is frequently made to the leveraging power of the Coalition, its ability to attract resources in support of both its own work and more broadly the cause of RH commodity security. For example in 2011 the Hewlett Foundation, UNFPA, IPPF, BMGF and the Advance Family Planning project contributed to the Access for All conference, LAC and SECONAF activities, and to grants for civil society members of the RMAWG. In the same year DFID allocated funds to the Concept Foundation’s Quality Reproductive Health Medicines project which is being administered under the auspices of the Coalition, and awarded $16m to the CARhs Group for commodity shortfalls. USAID also contributed $1m to this. The following graph shows stakeholder contributions from 2007 – 2011. Graph 1 Stakeholder contributions, 2007 - 2011 In a sense the Coalition is leveraging its own members. On the other hand none need have spent their money on Coalition related activities, and the donors at least could have spent outside RH commodity security entirely. The core investment by the donors to the Secretariat is effectively having a multiplier effect. The Secretariat glues the Coalition together, enables it to function, and in many cases facilitates, or at some point has had a key role in the efforts that attract the funding. This increases its value for money. 4.3.5 Generation of cost savings by the Coalition The most significant demonstration of value for money by the Coalition is the price reductions it has been able to negotiate for large orders. Communication between members who procure led to sharing of information on prices charged and combined negotiation with suppliers. The PGH has also been used in the case of Implanon to access committed donor funds more quickly. Reduced prices have been negotiated for Implanon (saving $7.3m to date Stakeholder Contributions 1,286,401 5,304,085 7,923,000 10,665,000 23,523,243 0 5,000,000 10,000,000 15,000,000 20,000,000 25,000,000 2007 2008 2009 2010 2011 25 Evaluation of the Reproductive Health Supplies Coalition and potentially $15.7 million if the 2012 order target is met), Jadelle (saving $7m to date and likely to be more) and the female condom (saving $359k). The RHSC has therefore saved at least $14m in commodity costs, and potentially $22m. Cost savings for those who procure also benefit those who manufacture: lower prices lead to more sales, greater volume, increased revenues. 26 Evaluation of the Reproductive Health Supplies Coalition 5.1 Current strategy The current Strategy (2007 – 2015) has served the Coalition well but is now somewhat out-of- date. Although the 2011 member consultation on the Strategic Plan concluded that it “is mainly on track and in line with current thinking”5, based on key informant input, the evaluators encourage that it be revised to reflect the expanded membership, current conditions, and future possibilities. The Coalition has been well guided by the three goals (see box 1 at the end of this section). In addition, the guiding principles elaborated in the 2009 Terms of Reference have also largely served the Coalition well, although more has been done to encompass method choice than the status of women and equity. There has been some drift from or lack of progress on the principle of country ownership in part because some RHSC operations (e.g. the functioning of the Working Groups, principle of volunteerism) have not enabled full participation and therefore ownership by countries. In fairness, membership criteria were restrictive early on and only recently has country-level representation increased to become the largest share among members. In contrast, principles relating to mutual accountability and functioning as a forum have largely gained momentum. The evaluation team detected reasonable congruence between the overall direction of Strategic Plan and the functioning of the Coalition over the last five years since the plan was written. (The RHSC Principles are discussed and analyzed further in section 6.1.1 of this Report.) In practice the ‘interactive overview’ matrix version of the Strategic Plan, which was developed by the Director to facilitate communicating it internally and externally, has probably served as a better strategic guide than the Plan itself. Although there was some criticism of having ‘dumbed down’ the strategy, the matrix presentation is a concise and easy way to understand the Strategy, and likely has more relevance to members than the lengthy document itself. However it is now rather out of date and there is a problem with the metrics (not specific to the matrix) as discussed in the following section. In past review documents such as the 2011 internal member survey, and in the course of this evaluation, various strategic issues have been raised. Among these are the role of the Coalition in addressing demand for commodities; the extent of the focus on country-level support to achieve Coalition goals; and whether the RHSC should address all RH supplies or mainly family planning supplies. In practice however these have not yet played out to create misalignment with the Strategic Plan nor have there been major tensions within the Coalition. There is now however a need to revisit the strategic goals, principles, and assumptions to address new challenges and opportunities and to enable some of the above issues and other strategic approaches to be considered for taking the Coalition forward. _________________________ 5 Kinzett, Steve (January 2011). Member Survey on the Strategic Plan, p9 5. To what extent is RHSC’s role adequately reflected in its strategy, theory of change and M&E framework? 27 Evaluation of the Reproductive Health Supplies Coalition 5.2 Theory of change A definition of a theory of change is given in Box 2. There is no such explicit theory of change within the RHSC to underpin the strategy. In various points of the Strategic Plan a theory is implied. One can draw links between the ”gaps, challenges and opportunities for meeting RH commodity needs”, the rationale for the Coalition, and the added value that participation in the Coalition is intended to bring. In practice the evaluators found that there was broad agreement among members on what the Coalition should be trying to achieve and how it should be trying to do that, again implying a common theory of change. However this is not articulated in one place. A process of developing a theory that is participatory (as far as is practical) should be used when the Strategic Plan is revisited. This would have the advantage that: � The strategy would be clearly based on achievable objectives, which is not the case currently. This is discussed below, section 5.3. � Members would ‘buy in’ to these objectives. � There would be clearer links between what the Coalition is doing and what is being measured. The absence of an articulated theory contributed to some problems with the MEF that are discussed below. � Risks and assumptions would be made more explicit. � Cohesion and common understanding among members of the role and potential of the Coalition would be further developed. The Coalition might find it useful to supplement its eventual theory of change with a logical framework. Log frames are useful for creating well-articulated links between overall goals and inputs. However, they can be problematic for example if used as a rigid monitoring tool. As with a theory of change, the process of developing one would help to resolve differences in stakeholders’ understandings about what is being done and why. 5.3 Monitoring and evaluation framework 5.3.1 Overview The Strategic Plan of 2007 contained a list of 58 illustrative indicators to suggest the kinds of information that could be used to monitor progress of the nine objectives under the three goals (see box 1). A monitoring and evaluation framework (MEF) was then developed, refining a list of 30 specific, operationalized and collectable indicators. These indicators had baseline values (mostly set for 2007), a strategy for collection, and a means of verification. It was also agreed that indicators should already be collected by one or more Coalition partners (or agency of a partner) and they should be relevant to 14 focus countries receiving targeted support by the Reproductive Health Interchange (RHI), Project RMA (Resource Mobilization Box 2: Explanation of theory of change “A theory of change (TOC) is a tool for developing solutions to complex social problems. A basic TOC explains how a group of early and intermediate accomplishments sets the stage for producing long- range results. A more complete TOC articulates the assumptions about the process through which change will occur and specifies the ways in which all of the required early and intermediate outcomes related to achieving the desired long-term change will be brought about and documented as they occur.” Adapted by Erin Harris, Research Analyst, Harvard Family Research Project, from Anderson, A. (2005). The community builder's approach to theory of change: A practical guide to theory and development. New York: The Aspen Institute Roundtable on Community Change 28 Evaluation of the Reproductive Health Supplies Coalition and Awareness), and the Global Programme of the United Nations Population Fund (UNFPA). The intention was to review the Strategic Plan and MEF in 2009 at the end of a grant from BMFG, and then again in 2015 to coincide with the end of the Strategic Plan. At the request of the evaluators, in 2012 the Secretariat kindly prepared another monitoring report to inform their judgements (see Annex 13). 5.3.2 Reporting to donors In addition to these occasional reports there are also annual narrative reports to the four donors compiled by the Secretariat and PATH. BMGF has one format and the DFID, USAID and UNFPA have a very similar one, but both contain effectively the same information. These report against Goal Three which covers the work of the Secretariat. In the report to BMGF the Secretariat reports against five objectives (different from those in the MEF) and 13 activities (see Box 1). These have remained unchanged since 2009, with the exception of a fifth objective and some associated activities in 2011 to accommodate the Hand to Hand Campaign. The reports provide a good summary of the work of the Secretariat and the system seems to work adequately. 5.3.3 Issues with the Monitoring and Evaluation framework There are neither adequate systems in place to monitor and report against Goals One and Two nor is the MEF measuring the right things. The scope of the MEF is too wide. The MEF is essentially trying to take the temperature of the state of the world’s commodity security while the Coalition can in practice only affect some aspects of it. As a result there is a mismatch between what the Coalition is doing and what is being monitored. For example Goal One, (Increase availability, predictability, and sustainability of financing for RH Supplies) is being addressed by the Coalition through the PGH, which has created added value out of existing resources, and the Hand to Hand campaign, which aims to generate new resources. However the indicators measure completely different things, such as how many focus countries are fulfilling their spending plans for RH supplies. It should also be noted that the focus countries no longer have great meaning within or relevance to the RHSC. These countries were originally selected on the basis of donor priorities, not Coalition aims. Some of these disconnects arise from lack of clarity about what the Coalition actually IS and what it DOES. On the one hand, the RHSC could be the sum of all its members, including some national governments, in which case an increase in expenditure on contraceptives by those governments could be said to be the work of the Coalition. On the other hand, it could be defined as simply the Secretariat, in which case only initiatives such as the Hand to Hand Campaign should be monitored. Neither position is completely correct. What the Coalition is and does has to lie somewhere in the middle, such that it can faithfully claim credit for work that its partners come together to implement. The interactive overview on the RHSC website comes closer to reflecting the actual work of the Coalition as linked to the Strategic Plan (or at least it did: it is rather out of date now). However it falls down when it comes to ‘how do we measure success?’ because it relies on the MEF indicators. A review of this website presentation makes the mismatch very transparent. 29 Evaluation of the Reproductive Health Supplies Coalition The MEF does not fully capture what the Coalition has achieved. Although the Coalition is undertaking some strategic initiatives which are very relevant for the achievement of the goals, they cannot all be reported against the MEF other than stating that they exist. Performance is captured through reports to the donors but that many key results do not appear in the MEF means that many of the achievements of the RHSC are difficult to establish without extensive document review and consultation. In addition the MEF does not capture regional achievements. The MEF is not used as a management tool. The evaluation team asked if another report against the MEF could be produced to contribute to this evaluation. This the Secretariat and its partners willingly did. However it was apparent that the existence or absence of an up-to- date monitoring report made little difference to the Secretariat, its members, or its donors. The donors’ needs are met by the annual report which details the work of the Secretariat against a separate set of objectives from those in the MEF. These annual reports also recount some of the wider Coalition achievements. Various other reports are provided to the Secretariat team on the progress of some of its initiatives, e.g. the CARhs team sends an annual report to the SSWG for review and discussion, and these prompt a response from Coalition members. However the MEF is neither a summation of all these reports, nor is it used as a way of assessing progress against its strategic goals or objectives, probably because no one finds it useful and it is a huge effort to gather the data. At the same time, the MEF with its broad scope is not used to assess commodity security more broadly, further diminishing its usefulness as a strategic tool. The MEF does not promote accountability. A well designed MEF should make it possible to link from goal to objective to activity onto workplans so that there is accountability for delivering against that MEF. However this is missing from this MEF. There may be value in some form of common monitoring of commodity security overall by the Coalition. An ongoing process of gathering data from multiple partners and tracking across time could be valuable for advocacy, resource mobilisation, and informing Coalition strategy. However, the added value of the Coalition undertaking this sort of exercise, beyond partners’ work in this area, would need to be considered carefully. In addition, this should be clearly separated from the MEF. The MEF needs a wholesale review. The three Goals are representative of what the RHSC is ultimately trying to achieve, but the objectives need to be reframed as outcomes (thus letting them sit as high level aims not to be achieved by the Coalition alone). In addition, new lower level objectives should be formed which have a closer relationship to the work of the RHSC and which imply accountability for their delivery. This would also enable a link to be formed between the new outcomes (the old objectives) under Goal three and the objectives against which donor reports are made and which at the moment are floating free of the MEF. This should also make it easier to translate the MEF into the workplans of the Secretariat and the Working Groups. All this will be made easier by developing a theory of change. Attention should also be given to trying to make data collection much less arduous and this should be considered when selecting indicators. The Executive Committee and the Director are aware of these challenges and the job description for the Monitoring and Evaluation Officer charges that person with addressing these issues. 30 Evaluation of the Reproductive Health Supplies Coalition Box 1: Goals and objectives for the RHSC Strategic Plan goals and objectives Goal 1: Increase availability, predictability, and sustainability of financing for RH Supplies. • Objective 1.A: Increase flow of new resources. • Objective 1.B: Ensure that countries and their governments make use of the resources already at their disposal. • Objective 1.C: Mobilize Total Market Resources Goal 2. Strengthen capacity of health systems to deliver RH supplies in a sustainable manner • Objective 2.A: Ensure the integrity of systems for supply chain management, both globally and at country level. • Objective 2.B: Provide a coordinated, short-term response to stock outs, supply disruptions and other failures that occasionally befall functional supply chains. • Objective 2.C: Promote effective delivery of RH health supplies in settings where routine systems for supply chain management have broken down or ceased functioning. Goal 3: Assure the added value of the Coalition as a productive and sustainable global partnership through efficiency, advocacy and innovation • Objective 3A. Foster greater harmonization among stakeholders at global and country level to generate economies of scale, maximize impacts, and minimize the risks of redundancy and confusion. • Objective 3B. Generate, disseminate, and facilitate the application of knowledge through innovation, research, and documentation. • Objective 3C. Offer a credible and sustainable institutional framework within which stakeholders can collaborate, build commitment to supply security, formulate strategies for change, develop common indicators of success, and measure progress against them. Annual monitoring objectives in the report to BMGF Objective 1: Ensure the functionality, smooth operation, and efficiency of the Coalition. Objective 2: Ensure that the Strategic Plan and associated program and policy frameworks (three-year work plan, M&E Framework, communications strategy) continue to shape and facilitate the Coalition’s work. Objective 3: Maximize the opportunities afforded by an expanded membership to further the Coalition’s effectiveness. Objective 4: Ensure the financial sustainability of the Secretariat and the Coalition. Objective 5: Support the Alliance for Reproductive, Maternal, and Newborn Health in contributing to the Coalition HANDtoHAND Campaign’s goal of reducing by 100 million the number of women with unmet need for contraception. 31 Evaluation of the Reproductive Health Supplies Coalition In this section, we review the structure and operation of the RHSC as it was and what it has become. The 2011 Julie Solo Report ably summarizes emergence of the global movement for reproductive health commodity security. Initiated by a dedicated group of activists and extended to others during the 2001 Istanbul Conference, the purposefully named Interim Group on Reproductive Health Commodity Security sustained this movement over the next several years. Midway through the decade the Interim Group gave way to formation of the RHSC. Interestingly, some of the same core components critical in the early phase remain central today: small groups of activists, ongoing discussions and deliberations among them, and global meetings to extend the movement to others. Building on what came before, the RHSC has evolved. Since 2006, under the overall guidance of an Executive Committee, a small Secretariat has been in place to support the membership and the three Working Groups. By 2009, RHSC had embraced extending membership to all organizations interested and aligned with the RHSC’s vision that “all people in LMI countries are able to access and use affordable, high-quality supplies, including a broad choice of contraceptives, to ensure better reproductive health.” By 2010 the funding base of the RHSC had expanded beyond the Gates Foundation, which had supported RHSC through its formative years, to a coalition of funders now including DFID, UNFPA, and USAID. 6.1 2009 Terms of Reference The 2009 Terms of Reference (TOR) contains a set of guiding principles and a diagram of the structure within which the RHSC operates. As such, the TOR provides a relevant and useful roadmap to review how the RHSC has developed and expanded over the years. 6.1.1 Guiding Principles of the RHSC The RHSC’s guiding principles contained within its 2009 Terms of Reference (TOR) derive in large part from the set of best practices for global partnerships for health. In 2005, the High Level Forum (HLF) on the Health MDGs set out principles for best practice in global health partnerships, Building on the Paris Declaration on Aid Effectiveness and remaining useful today, these best practices are summarized as follows: � Ownership: Global Health Partnerships respect partner country leadership and help strengthen their capacity to exercise it � Alignment: Global Health Partnerships base their overall support on partner countries’ national development strategies, institutions and procedures � Harmonization: Global Health Partnerships’ actions are more harmonized, transparent and collectively effective; Global Health Partnerships collaborate at the global level with other partners to address cross-cutting challenges such as health system strengthening � Managing for Results: Global Health Partnerships work with countries to adopt and strengthen national results-based management 6. In what ways and to what effect has the structure and operation of RHSC evolved, particularly since 2009? 32 Evaluation of the Reproductive Health Supplies Coalition � Accountability: Global Health Partnerships provide timely, clear and comprehensive information6 The 2009 TOR lists seven guiding principles as critical for “achieving a sustained supply of affordable, quality reproductive health supplies in low and middle income countries”. Summarized from the TOR, how each has since been applied is reviewed below: � “The RHSC understands that achieving (its) vision requires (optimally applying) public and private resources…and utilizing a multi-stakeholder approach to (serve people’s RH supply needs and) improve RH behaviors” – The RHSC has become the largest extant body dedicated to FP/RH. Opening membership after 2009 to all organizations aligned with its mission “to ensure that every person is able to obtain and use RH supplies” has made this possible. Currently approaching 200, RHSC member organizations cover a broad range: small country-level civil society organizations and large commercial manufacturers; private foundations; government donor agencies; and multi-lateral donors; developing country governments and social marketing enterprises; supply system expert groups and generic producers of particular products. � “The RHSC acknowledges that (effectiveness results through) complement(ing) the actions of its individual members (through leveraged, collective action; promoting collaboration and cooperation); and work(ing) by consensus towards common goals” – Structurally, the RHSC is well positioned to promote collaboration and combined action among its members. Practically, however, much remains to be achieved. The membership has grown beyond and not been sufficiently integrated within the original structures. As a result, many opportunities for collective action towards common goals are missed. � “The RHSC believes that increased country ownership is fundamental…(and) will work through national governments to develop supportive policies, plans, resource commitments, and capacities” – Country ownership for the RHSC remains a principle not a practice. While membership at country levels has increased, regular involvement within RHSC structures has not. Furthermore, the few governments approached – as part of holding RHSC annual meetings in Uganda and Ethiopia – remain exceptional, not continual. � “The RHSC recognizes the role of both industrialized and developing-country manufacturers, for planning, providing, and delivering quality RH supplies” – The RHSC has not only recognized but now also encompasses manufacturers within the Coalition. Three major pharmaceutical companies have joined the RHSC as well as a number of generic manufacturers. This opens the way toward private-public sector dialogue, edifying for both sides. � “The RHSC strives to ensure greater equity and improve access to RH supplies for those who are poor and marginalized” – This remains the North Star for many Coalition members. Some global improvements, such as stock-out alerts and better data on procurements, eventually reach such populations. However, efforts to more directly impact those in need have been few within RHSC. Especially within the broadened perspectives of private, commercial sector engagement within the RHSC, this is an important principle to retain. � “The RHSC recognizes that (diversity of) needs (requires making) a range of products (available within a client-centered approach, including bringing) attention to underused RH options and the research needed (to meet) contraceptive method gaps” – Member concerns with these issues have surfaced and received some focused attention, particularly within the Caucus on Under-used Methods, the Task Force for Maternal Health _________________________ 6 Best Practice Principles for Global Health Partnership Activities at Country Level, High Level Forum on Health MDGs, 14-15 November, 2005, Paris 33 Evaluation of the Reproductive Health Supplies Coalition Supplies, and the MDA Working Group’s recent outreach to reproductive health product developers. � “The RHSC recognizes (that disadvantages in) women’s status, resources, and decision- making (undermine RH commodity security)” – This remains central within a movement originated, as Julie Solo reminds us, “(by) a small group of committed and driven women”. Especially within a structure that lends itself more to technical discussions than practical solutions is retaining this principle important. 6.2 Operational Structure of the RHSC RHSC is an operational entity that is subsumed under PATH, which receives and manages funds designated for the RHSC. The decision to not officially register RHSC is practical as well as purposeful. Moving the funding through PATH was useful for the Gates Foundation, its founding supporter, which already had an established funding mechanism with PATH. As other funders joined to provide collaborative funding for RHSC, they have continued to channel funds through PATH. With PATH providing backup management and administrative support, particularly through the start-up years, RHSC was able to quickly take form from 2006 onwards. With guidance from the Executive Committee, staff members were soon recruited and the Secretariat established in Brussels. The structure within which the RHSC works to support Coalition members and the work on commodity security is detailed in the following. 6.2.1 Executive Committee The Executive Committee is the governing board of the RHSC, a committee rather than a board because the RHSC is not an officially registered organization. In scheduled bi-annual meetings, the Executive Committee reviews past accomplishments, assesses the present situation, and sets future directions for the coming year. The Executive Committee includes all four funders of the RHSC, heads of the three Working Groups, and representatives of organizations with notable presence in international commodity security and family planning, namely the International Planned Parenthood Federation (IPPF), the Pan American and West African Health Organizations (PAHO and WAHO), and the World Bank. Executive Committee members, including the funders themselves, noted that the donor representatives form an inner core and exercise correspondingly greater influence within the Executive Committee. (An exception is the World Bank, which has not been notably active in the RHSC since serving as the Executive Committee’s first Chair.) Chairpersons in particular, historically also from the donor community, have been particularly influential in shaping the RHSC. Though this risks becoming a donor-driven entity, to date this has been ably counterbalanced by recognizing that the RHSC is a coalition of its members. Key informants at various levels encouraged expanded representation on the Executive Committee. Among specific recommendations offered were to include new categories representing member constituencies, such as manufactures and country-level organizations. In addition, one well-placed key informant suggested deepening discussions and deliberations within the Executive Committee by adding independent experts on commodity security and international family planning. 6.2.2 Secretariat The Secretariat is the operational center of the RHSC. The RHSC’s group of core staff has turned over some since 2006, but remains small and focused on supporting the Coalition 34 Evaluation of the Reproductive Health Supplies Coalition members. Secretariat staff members give special attention to the established structure of the three Working Groups. In addition, Secretariat staff support the Caucus and the Task Force, which have emerged as venues to represent and express member interest in particular areas. That the Secretariat has operated under the same Director since 2006 has provided consistency even as the RHSC has grown. 6.2.3 Three Working Groups The three Working Groups are designated structures of the RHSC through which Coalition members engage particular substantive issues: Systems Strengthening; Market Development Approaches; and Resource Mobilization and Awareness. Enshrined in RHSC founding documents, all three Working Groups built on work already initiated under other projects. For example, Deliver as USAID’s lead organization for strengthening supply chains has been involved in efforts to improve FP commodity data collection and dissemination well before the RHSC came into being, initiating early versions of the RHInterchange as well as the CARhs. Similarly, the Interim Committee on Reproductive Health Commodity Security , funded by the Gates Foundation from 2003 to 2006, was a precursor not only to the RHSC generally but also to the Resource Mobilization and Awareness (RMA) Working Group. Earlier interest in market segmentation for tiered pricing of contraceptive commodities provided precedent and momentum for the more broadly conceived Market Development Approaches Working Group, which focuses on equity as well as access. During the early, initial years of the RHSC, member organizations were well represented within each of the three Working Groups. However, with recent increases in RHSC members, the Working Groups have not correspondingly expanded their reach. The Working Groups thus risk becoming more an elite sub-culture of founding members than a vital and vibrant part of expanding interests and possibilities. If only by historical precedent, the primacy of the three established Working Groups seems largely unchallenged. An evolving RHSC, embracing its broader membership and making strategic adjustments, may need to reconsider the current structure and adapt to serve evolving needs. 6.3 RHSC Flexibility and Adaptations Beyond the Working Groups, members have found need for and formed other structures to pursue particular interests. This clear evidence that the Coalition is member-led also demonstrates the Secretariat’s role and willingness “to lead from behind”. However, like the Working Groups, such structures as the Caucus for New and Underutilized Methods and the Task Force for Maternal Health Supplies have not yet been an avenue to expand country- level participation. The Regional Forums in contrast have potential for wide as well as deep involvement at country level. However, the list of regional forums which never made substantial contributions at country or other levels is long. These various RHSC structures, all of which receive support from designated Secretariat staff, are briefly described below: 6.3.1 Caucus for Underutilized Methods Interest among a few members in giving special attention to certain contraceptives and other RH products led to formation of the Caucus on New and Underutilized Methods. Discussions beginning as early as 2006 eventually led to forming a group under the umbrella of the RHSC. Different from Working Groups, the Caucus did not operate within work streams or develop an annual work plan. Instead, the Caucus meets less frequently, has regular conference calls, and is intentional about trying to influence the three Working Groups. The Secretariat saw value in having this added focus and organized to provide backup support for Caucus discussions and its annual face-to-face meetings. 35 Evaluation of the Reproductive Health Supplies Coalition 6.3.2 Task Force for Maternal Health Supplies Expanding RHSC beyond contraceptive commodities similarly grew out of member interest, including discussions that occurred initially within the Caucus. Focus on maternal health gained impetus during the 2011 Annual Meeting in Addis. Encouraged by a participating representative of a private foundation with longtime strategic investments in maternal health, a special lunch roundtable was convened to discuss maternal health supplies. Building on this expression of member interest, the Task Force soon emerged. Deciding to focus upon three critical commodities – magnesium sulfate; oxcytocin; and misoprostal – gave the Task Force a unique position among other groups outside the RHSC similarly working on maternal health. As with the Caucus, the RHSC quickly recognized the value in this and assigned a Secretariat staff member to support Task Force operations. 6.3.3 Regional Forum for Latin America and the Caribbean (LAC Forum) Interest in bringing the LAC Forum into the RHSC emerged out of the regional meeting in Panama in 2010. As one key informant who was there observed, “great diplomatic skills” of the RHSC Director helped to make this happen. As a donor representative noted, occurring at a time when donors were withdrawing from the LAC Region, the Forum is a useful way for countries in that Region to stay connected with the field of commodity security. Similar to the RHSC itself, international organizations appear to be the drivers within the LAC Forum. UNFPA and USAID have made some funding available for the LAC Forum to operate its own Innovation Fund in support of regional Working Groups. The most recent RHSC newsletter listed the first round of awards made. 6.3.4 Francophone Africa Forum (SECONAF) Similar to what transpired in the 2010 Panama Meeting of LAC, a recent regional meeting in Burkina Faso was pivotal for bringing the Francophone Africa regional forum, SECONAF, under the umbrella of the RHSC. Again, central leadership of the RHSC had a key role in bringing this about. The subsequent decision by the Hewlett Foundation to consider making a grant through the RHSC for SECONAF should create some momentum to build upon. RHSC’s courting of the French government, which will host the next annual RHSC annual meeting, could also prove strategic and useful. 6.4 Support for Structures and Operations 6.4.1 Secretariat Staff Changes (2010-2011) The Secretariat experienced some staff turnover and consequent disruption during the last year. Heavy staff workloads, including tremendous effort involved in backstopping RHSC activities and events, such as the 2011 Annual Meeting in Addis, may have taken some toll. The Secretariat has come through these changes, but in the process lost some technical expertise. A number of key informants remarked that greater technical expertise would usefully add to the current configuration of the Secretariat. 6.4.2 New Staff Positions Two new staff positions are being filled within the Secretariat. One is already in place and handling communications within the RHSC, which had until recently been outsourced to a part time consultant and supported in-house by PATH. The second position will be for Monitoring and Evaluation, recognizing that RHSC has not given this consistent attention over time. 36 Evaluation of the Reproductive Health Supplies Coalition 6.4.3 Working Group Leader Turnover and Secretariat Support Though not explicitly stated as operational policy, the Secretariat has encouraged leadership turnover within each of the three Working Groups. In 2012, a convening by the Secretariat of Working Group heads encouraged that each Group undertake a visioning exercise to confirm or refocus direction for the Group. Sensing that Working Groups were not as vitally involved in the mission of the RHSC as they once were, the Secretariat has sought to reinvigorate them. Particularly during these Working Group transitions, back-up support provided by designated Secretariat staff is crucial. As expressed by many key informants, this support from the Secretariat is also greatly appreciated. Though the relevance of work streams received attention in subsequent SSWG or the MDAWG meetings, focused attention to conduct a visioning exercise did not occur within either. Consultation with and assistance from professional meeting facilitators would usefully contribute to such RHSC convenings, including annual meetings. More broadly, and particularly within the context of updating its Strategic Plan, the RHSC needs to consider how these established structures will serve the new and emerging opportunities provided by a vastly expanded membership. 6.4.4 PATH Support for the Secretariat PATH continues to provide back-office support to the operations of the Secretariat. This remains primarily within the areas of financial and human resources management. PATH has also been able to assist in completing particular RHSC tasks when requested. As both a member of and fiscal agent for the RHSC, PATH needs to continue to avoid conflict of interest areas. 6.5 Future Secretariat Needs Keeping the RHSC as several termed it “lean and mean” appears highly prized. Many key informants were pleased and surprised that the Secretariat is able to do so much so well with so few. Furthermore, key informants, and particularly donors, expressed interest in keeping the RHSC trim, even while several saw need to add some technical depth to the Secretariat . Some key informants, including within the Secretariat itself, recommended adding a deputy director. Other key informants suggested that expertise on policy, advocacy, and country-level programs would be useful additions. Many key informants attested to the usefulness of the Innovation Fund and some suggested making this or similar funds, as is now occurring on a limited scale with remaining Innovation Funds, also available for selected country member organizations. Current operations and structures of the RHSC are further assessed in the next two sections, Section 7 and 8. 37 Evaluation of the Reproductive Health Supplies Coalition The evaluators visited Brussels, spending the better part of four working days within the RHSC office. In the process, they logged more than fifteen hours interviewing Secretariat staff, including over ten hours with the Director. A follow-up interview with the Director took place in London and the evaluators also interviewed the half-time RHSC staff member based in PATH/Washington DC. Three staff members of PATH/Seattle, the fiscal agent, RHSC’s administrative backstop, and a Working Group member, were also interviewed. In addition to materials available on the RHSC website, the evaluators also requested and received from both Brussels and Seattle a number of documents relevant to RHSC structure and operations. This information, together with member responses to questions about the efficiency and effectiveness of RHSC operations, provide the basis for assessing what has worked well and what should be improved within RHSC. 7.1 Strengths within RHSC Internal Operations 7.1.1 RHSC Director As Director, John Skibiak received a great deal of praise from Coalition members interviewed. Almost all key informants offered high compliments about his work and what he has been able to accomplish. One key informant from a large international NGO said: “The director, John Skibiak is a strong person…since he’s been on the job he has learned a lot and increased his network and he has done what from my distance seems a very good job”. Another offered this perspective: “John Skibiak…is a very good person to (head the RHSC) because he’s broad and can work with a large set of people in a way that is helpful for bringing in different perspectives”. And from another: “John that's been the driving force behind (raising the prominence of FP/RH commodity security)…he's got involved in a whole range of forums which we never used to get involved in. But the problem is he has a limitation – he’s one man!” One key informant more expansively observed the following about the RHSC Director’s evolving leadership: “In the last 24 months, the Coalition…as an entity, in other words John himself, has…felt confident enough and has had the opportunity to get out and about as the Executive Director of the Coalition, with the feeling of strength and begin to engage at the international level, you know, at the table as the Executive Director of the Coalition. I think it is a relatively new phenomenon, but a result of the support that he has been given by the key stakeholders of the Coalition, who feel that supporting the Coalition and working through the Coalition is one of the key ways in which they will meet their own internal strategies for reproductive health.” Assessing the Director’s leadership and what he has done with the resources available to take the Coalition in interesting new directions, the evaluators conclude that he has been just the right person to lead the RHSC through this period of rapid membership growth. If any criticism can be levied, and it is one against the Executive Committee as well, it is that too much has been taken on without sufficient time and attention to ensure alignment with the 7. What works well and what needs to be improved or changed about RHSC’s internal structure and external operations? 38 Evaluation of the Reproductive Health Supplies Coalition RHSC strategy and members. As a private foundation staff member, herself a long-time participant, noted: “On the (RHSC) strategy, we do have a strategy but… 2007 is when it was developed and it's more aspirational … it doesn’t reflect the current need, and this push and pull between how (the Secretariat) can just, you know, help to orchestrate all their members and let them lead, or (that it) need(s) to take leadership in (some) areas.” Especially during the last few years has the RHSC been moving quickly forward, and it has done so with an outdated strategy that reflects neither changes that have occurred nor new opportunities that have opened. Keeping followers and future directions aligned is a leader’s ultimate responsibility. In contrast, as reviewed in the previous section (Section 6) on how the RHSC has evolved, the guiding principles set forth in the 2009 Terms of Reference are less time-bound and still well able to represent how the Coalition aspires to operate. 7.1.2 The Secretariat Key informants perceived Secretariat staff members, including the Director, as dedicated, supportive, and hard working. Many respondents expressed great appreciation for how much so few are able to do. Several expressed surprise that the core staff was as small as it is: four based at the Secretariat in Brussels and one half-time staff member in Washington DC. (Within the last month, the Brussels-based Secretariat added a communications staff member; an M&E specialist is also in the process of joining the Secretariat.) Without exception, key informants in direct contact with Secretariat staff were complimentary about help received and relationships developed. Organized to assist the Coalition Working Groups and other structures (i.e. the Caucus, Task Force and the two regional forums), this small staff is well organized, highly motivated, and over-worked. Some evolution in the amount and level of support provided is occurring, from being responsive to what a particular group may need or request, to being more proactive to ensure that a particular group moves along. As such, Secretariat staff members assigned to Working Groups have begun to function as co-leaders when needed, though clearly deferring to the designated and elected group leader. 7.1.3 RHSC Structures The various groups, and in particular the RHSC’s formal structure of Working Groups, have over the years experienced ebbs and flows. Based on voluntary participation by members, all of whom have other fulltime responsibilities within their own organizations, Working Group members and leaders alike report being stagnated at times and more engaged at others. In contrast, both regional forums are relatively new and have not as yet experienced such ups and downs. The Task Force, which came out of discussions among Caucus members and during the 2011 annual meeting in Addis, is also new. The Caucus, which serves a cross- group function as well as leadership on a particular issue, seems to maintain a sustained level of interest in new and underused methods. Overall, the willingness of members to sustain such groups is a measure of the value they gain from participation. 7.1.4 RHSC Donor Support Several private foundations, among them the Gates Foundation, the Packard Foundation, and the Wallace Global Fund, supported the movement to improve contraceptive commodity security during its formative years, 2000-2005. The Gates Foundation continued its support enabling the RHSC to emerge to sustain this movement. In 2009, three other donors joined 39 Evaluation of the Reproductive Health Supplies Coalition with Gates to continue funding for RHSC through 2012. (In addition, the Gates Foundation has notably stepped in when needed with supplementary support to establish the Innovation Fund and to expand the Addis Meeting.) The four donors have been purposively penurious in supporting the RHSC. A small Secretariat, “lean and mean” as several stated, seemed to make sense for supporting a “coalition of the willing”. Furthermore, the RHSC was not envisioned as a vehicle for making funds available. As the Director stated, when asked if the RHSC should have substantially larger amounts of funding to spread among partner organizations, “What you wouldn’t want is a pot of money so that organizations now feel they have a feeding trough to fund their own activities that they would have done anyway.” Relatively small amounts provided through the Innovation Fund apart, the RHSC has not aspired to be a Global Fund or Global Programme. Rather it works through what member organizations are able to contribute by virtue of mutual interest in particular subject areas and what members may be able to attract in kind from within their own organizations and/or in funding from other donors. This is what has worked in the past. Whether this same model is the one to carry into the future is discussed under the Questions addressed in Sections 9 and 10. 7.2 Weaknesses within RHSC Internal Operations 7.2.1 Small staff, growing membership As indicated above, the dedicated staff of the Secretariat is stretched thin. The lean could well become mean as the pace continues and burnout occurs. Staff recently added for communications and M&E will not reduce significantly the work burdens of current staff, since related work that was previously out-sourced (largely to PATH) will now be handled within the Secretariat itself. An important addendum to the operations and structure of the RHSC is that volunteerism has limits. Considerable burden falls in particular on those who lead the Working Groups. In addition, effective action along designated work streams of Working Groups often requires considerable effort beyond an individual’s normal scope. As a result, exciting ideas, projects, and studies generated within Working Groups sometimes, if not often, fall by the wayside. The Innovation Fund has, in this situation, proven a useful mechanism by making some additional resources available for particular activities. 7.2.2 Structural Shortcomings Another constraint is that the large numbers and especially many newer members have not meaningfully contributed through the Working Groups and will likely not be able to do so in the future. Even without the added complications of geographical and financial distances, the existing structure of three Working Groups could not absorb or deal with vastly expanded participation. Indeed, most Working Group respondents indicated that they have between 15 and 30 members who participate somewhat regularly, with an inner core of 4-6 who are highly involved and dedicated in discussing and moving particular subject areas and agendas forward. In addition, there are other considerations with volunteerism. What participation contributes to individuals is as important as what members contribute. Respondents to the member survey undertaken as part of this evaluation provide some insights on involvement in the Coalition. The total number of responses to the electronic survey of 12 questions was 65, a self- 40 Evaluation of the Reproductive Health Supplies Coalition selected one-third of all the RHSC organizations to which the survey was sent. This is itself a reasonably high response rate for a survey of this kind, especially considering that more respondents reported only moderate levels of active participation (at least once a quarter) than any other category. Furthermore, more than one-third of respondents reported that they had no affiliation with any RHSC group. Though the majority of respondents work within INGOs, fully one-third reported that they were part of country-level organizations. Despite being far removed from active participation, many seem to gain from being a part of a broader movement. (A Summary Report of the Member Survey is in Annex 5.) However, it may well be a missed opportunity that current RHSC structures do not actively engage more member interest, particularly those who are at country level. Similarly, the large international organizations, many of which have been with the RHSC since its founding, do not themselves sufficiently involve staff working at ground level, except for sponsoring some to attend an annual meeting. As a result, the Working Groups continue to function as they always have - at a high, largely headquarters level, focused on global concerns and advances. Encouragingly, the recent MDA Working Group meeting made a concerted and largely successful effort to reach beyond its usual participants. Practical implementation at country level does occasionally occur as a Working Group activity, usually with Innovation Fund support. However, this is driven more by a global perspective on what is needed to advance the field than practical consideration of what would benefit a particular country. Finding opportunity for country-level input to move commodity security forward would usefully add to what the RHSC Working Groups are able to contribute. In addition and alternatively, new structures focused on particular countries or groups of countries (the latter to facilitate cross-country interchange and learning) may be another operational approach to consider. It may also be that the model of the “Working Groups as the engines of the Coalition” is itself outdated and outmoded. With leadership being sometimes exercised from the top and the convener or facilitator role not always clear, perhaps an entirely different structure for the RHSC is needed to continue operations. This is a matter better left to the members in subsequent discussions as to who they are now, where they want to go together, and how they may get there. In other words, when they next update and revise their Strategic Plan and when they logically link RHSC and member inputs with outcomes expected. 7.3 External Operations The RHSC, most notably through the Director and especially over the last two years, sometimes goes beyond the existing structure to investigate new areas of potential engagement. As occurred with contacts and discussions with several manufacturers, this may result in new members and perspectives within the RHSC. Such prospecting was also instrumental to bringing the two regional forums under the umbrella of the RHSC. On other occasions, such as a recent meeting in Bellagio on long-acting reversible contraceptives for developing countries, co-sponsored with the Population Council and the Federation of International Gynecologists and Obstetricians, the RHSC increases its visibility and possibly its reach into new areas of substantive inquiry. Remaining informed about perspectives and understandings not currently receiving attention within the RHSC will help guide future operations. 7.3.1 Contribution and Attribution beyond RHSC It is often difficult to separate out the contributions of member organizations from those of the RHSC. Clearly the RHSC enters a gray area when it claims some achievements of members 41 Evaluation of the Reproductive Health Supplies Coalition as at least partly its own. Members themselves were sometimes divided as to which organization should get credit for what. For example, some key informants thought that RHI and CARhs would have happened independently of the RHSC. Others gave credit to RHSC for extending and publicizing these same initiatives. One key informant, a Working Group leader, suggested that such contributions exist along a continuum ranging from spawning or germinating an idea, to actually developing it (usually done by a single organization), to promoting and spreading its usage, to providing some ongoing maintenance. In the case of the Pledge Guarantee for Health, this respondent viewed RHSC as having made contributions from the earliest stage. In the cases of RHI and CARhs, RHSC was on the other end of the continuum, encouraging their use and ongoing maintenance. Attribution is similarly murky with regard to funding claimed as leveraged by the RHSC. In some cases, key informants who were directly involved, such as for AccessRH and Pre- Qualification thought that RHSC was instrumental for their organizations to attract funds to take initial Innovation Fund studies to scale. Others offered more skeptical assessments akin to a faulty syllogism that organizations routinely count as leveraged everything within reach. In some areas however what was birthed within the RHSC may have wide-ranging impact. For example, AccessRH, especially if combined with RHInterchange, may one day greatly facilitate commodity ordering and procurement. The Caucus, as another example, influenced the UN Commission on Life Saving Commodities for Women and Children to include family planning broadly and three contraceptives in particular among 13 essential commodities. The Task Force was similarly influential in having its three priority commodities included within the UN Commission’s list. Eventually such efforts will have impact all the way to country level. However, though country-level Coalition membership has increased, within the current structure the RHSC has yet to build much upon this growing reach. To do so, RHSC will need to re-strategize and re-focus to place some needed priority on country-level engagement and participation. 7.4 Appropriateness of RHSC Structures and Processes The internal structures of the RHSC were set out in the 2009 Terms of Reference. The structural core of the RHSC – three Working Groups supported by designated Secretariat staff – remains largely what it was when the Coalition started: the seed bed for connections among expert staff in well-funded organizations. Clearly there has been some evolution and adaptation since that time. For example, the Caucus emerged in response to member- generated interest in focusing on under-utilized methods. Its continuing existence testifies to RHSC’s adaptability as well as efforts to cross-connect the Working Groups. Similarly, the Task Force on Maternal Health Supplies also emerged in response to member-expressed need, most visibly at the Access for All Conference in Addis. 7.4.1 RHSC Working Groups The Working Groups have been operationally idiosyncratic with varying effectiveness. Each has its own unique character of participants. For example, the Systems Strengthening Working Group (SSWG) continues as the work space for the original core of logistics experts, many from or affiliated with USAID, UNFPA, and JSI. The Market Development Approach Working Group (MDAWG) has been the most explorative of the groups, reaching out most recently to learn about promising new products and possibly influencing how they may come to market. The RMAWG comprised of advocates has been consistently most concerned with moving beyond commodities to change policies and have an impact at country level. 42 Evaluation of the Reproductive Health Supplies Coalition All three Working Groups seem have gone through periods of high involvement as well as stagnation. This is not unrelated to the availability of funds to actually implement activities. For the RMAWG, Project RMA initially and Advance Family Planning subsequently provided funding to implement advocacy efforts, notably at country level. Similarly, Deliver’s active presence in the Systems Strengthening Working Group provided opportunity for in-kind contributions from this well-funded Project. Without such supplementary support, for amounts and time frames beyond what the Innovation Fund could provide, the Working Groups may have long since faltered. After several years under the same leaders, the three Working Groups similarly experienced some loss of momentum and energy. This is being addressed by transitioning Working Group leadership positions as well as repositioning support from the Secretariat. Two of the three Working Groups are now operating under new leaders, with the third soon to change as well. Secretariat staff members are attentive to providing what the Working Groups and their leaders need. In the interest of more closely aligning activities with Working Group strategies, the Secretariat has recently encouraged that visioning exercises be conducted within the different Groups. Support from professional facilitators to plan and run RHSC meetings could add value to Working Groups as well as other Coalition convenings. 7.4.2 Effectiveness of Innovation Fund The Innovation Fund began in 2008 with support from the Gates Foundation, which allocated $2.0 million to support RHSC Working Group generated studies and projects. To date, the review committee (comprised of selected Executive Committee, Secretariat, and PATH staff members) has approved 16 awards. With funding in the range of $50,000 to $200,000, successful applications from the three Working Groups included studies of new areas for commodity security advancement, such as pre-qualification; applications of specific logistics and market analysis tools within country settings, such as applying Total Market Initiative, a market segmentation tool, in Honduras and Madagascar; and production of the documentary Empty Handed on the importance of contraceptive logistics. A summary assessment of results at country level is at Annex 9 For the most part, the Innovation Fund has not been truly innovative nor has it approved proposals from far beyond the usual suspects. As several respondents expressed concerns about the Innovation fund, it has been misnamed and needs to be renamed. However, it has had the useful function of focusing Working Groups on particular subjects of interest and providing some funding to cover the costs of implementation. Without such supplementary funding, the Working Groups may have lost impetus and accomplished far less than otherwise. The Innovation Fund is further considered in the next section of this report, which considers partner-led and pro-active dimensions of the RHSC. 43 Evaluation of the Reproductive Health Supplies Coalition The RHSC builds on member interest in and contributions (in kind or through leveraged funding) that members may make to improve FP/RH commodity logistics and security. This “coalition of the willing” has built an organizational architecture that facilitates “leading from behind” or “leading from within”. However, the RHSC is not simply the sum of its membership structure, an accumulation of partner-led initiatives. While the RHSC Secretariat serves an increasingly diverse membership, over time it has also become selectively proactive, moving towards and sometimes advancing entirely new opportunities for commodity security. How this has come about and the balance maintained between member-led and proactive is pursued in this section. 8.1 Partner-Led Initiatives within RHSC Many Coalition initiatives exemplify a member-driven and directed RHSC. The Working Group structure, as codified in the 2009 Terms of Reference, builds upon the RHSC’s commitment “to be participant driven in its strategy, implementation, and governance…and to working collaboratively and effectively to…resolve supply problems and ensure long-term RH supply”. Key informants often referred to Working Groups as the “engine that drives the Coalition”. The following provide specifics on how, when, and with what results partner-led has come to characterize the RHSC. 8.1.1 Partner contributions to RHSC “Coalition of the willing” well defines the external as well as the internal operations of the RHSC. Members self-select into the RHSC, whose only requirement is that organizations be duly registered. (Individuals are not allowed to join.) Joining the RHSC implies some alignment with its overall mission and vision. As part of orientation, new members are asked to affiliate with one of the three Working Groups. However, there are no expectations or obligations as to how much involvement there will be or what this will entail. As several key informants noted, new members often only participate in a single Working Group meeting, usually one held in conjunction within an annual meeting. Within the Working Groups and other member structures, clearly some individuals and organizations participate more than others. The discussions, deliberations, and decisions about what to work on are often useful for advancing some particular aspect of FP/RH. However, who participates and to what extent is a matter of willingness, interest, and also opportunity. For some, the structure works well. Those more distanced from or practically challenged by the level and substance of Working Groups, participation in these internal structures is not likely to happen. “Coalition of the willing” effectively defines an inner coalition comprised of those who are most able to be and already are well-connected. As one key informant stated, “a coalition of the willing…(where) the work gets done by all the membership; that's all right up to a point, but you know, it’s not my day job.” 8. To what extent has RHSC been responsive to partner-led initiatives or proactive towards new opportunities? 44 Evaluation of the Reproductive Health Supplies Coalition 8.1.2 Innovation Fund Working Group projects and studies receiving support from the Innovation Fund were clearly partner-led. Sometimes these involved collaboration among member organizations, often for activities that would not otherwise have been implemented. The Secretariat’s internal review of the Innovation Fund, conducted in late 2010 by surveying members, forthrightly exposed several shortcomings and weaknesses. Though the Fund served some useful purposes, a number of administrative as well as substantive improvements were indicated. Among these were needs for greater clarity about how the Fund furthers the RHSC Strategy and Working Group operations, need for improved review criteria and processes, and need for follow-up or further application once a project or study is completed. In addition, no awards were made for proposals emanating from country-level. To correct for this, funds remaining have been set aside for applications from in-country organizations. To assist with proposal development, the RHSC has given a small amount of support to Population Action International. The evaluators agree with these conclusions of the internal review and the effort to make some country-level awards. The Innovation Fund has also fallen short on what could have been a hallmark of these awards: fostering collaboration among member organizations. Almost all awards involved projects and studies conducted by a single implementing organization. Left to their own and despite the connections that occur in the Working Groups, some organizations may seek to further their own interests and agendas than to more broadly contribute to the field. Proposal criteria that prioritize connections and collaborations across member organizations, including those at county-level, should receive greater weight. What should happen to the Innovation Fund going forward? The Innovation Fund has already supported 16 projects and studies of interest to Working Group members and is also being replicated within the LAC Forum. The small amount remaining in the Fund is currently being directed towards in-country organizations, a useful effort to support country participation. As a respondent to the member survey remarked, “(Being a Coalition member provided) opportunity to compete for funding though we never got any, but is another chance to learn from our shortcomings, as we look for funds to implement our activities“. Going forward, without a replenished Fund, Working Groups will very likely lose momentum. However, as several key informants observed, for the most part awards have largely not been innovative. Some suggested renaming or reorienting the Fund. Perhaps a Country Catalyst Fund, with weight placed on collaborative work at country level, would be a useful adjustment. 8.1.3 Partner-Led Initiatives: Caucus and Task Force From early on, RHSC members were clearly empowered to organize time and effort around supply initiatives and commodity security concerns. Annual Meetings often provide opportunities to involve partners in leading various sessions. For the most part, however, partner-led occurred within the Working Groups. In addition, when interest emerged and coalesced around particular subjects needing sustained attention and effort, members themselves created new structures outside the Working Groups. The Secretariat, recognizing the importance of such partner-led initiatives, quickly organized staff resources to provide some administrative and backup support for such nascent structures. Discussions and deliberations among members are the seedbeds for such partner-formulated structures. The Caucus on New and Underutilized Reproductive Health Technologies and the Task Force on Maternal Health Supplies are two member-initiated structures that took shape in this way and continue to work under the umbrella of the RHSC. 45 Evaluation of the Reproductive Health Supplies Coalition As early as 2006, around the same time that the RHSC itself was taking shape, two Coalition members initiated discussions for what eventually became the Caucus on New and Underused Methods. As one key informant explained, “(the Caucus) didn’t fit anywhere really . there’s a bit of resource mobilisation, there’s a bit market development, there was a bit of SSWG, so it sort of went across.” Another who was involved from the beginning described how the Caucus came to be accepted: “The (RHSC) was flexible enough to say ‘OK here is a need and some of our members think this is important’…we didn’t really ask for permission. We said we’re going to do this but what we want is for the (RHSC) to pay attention to us. So this is part of the membership and we framed it as a membership initiated activity…So you begin to see how these new themes come in because of a decision to open things up.” Attention for maternal health supplies actually arose first within the Caucus and then gained momentum at the 2011 Annual RHSC Meeting. A key informant, who participated in a roundtable lunch in Addis on maternal health supplies, described what happened in the aftermath: “There was all this real interest and obvious momentum, so we decided to form a task force that would talk about how the Coalition could add value to efforts to address maternal health supplies”. In follow-on discussions, the Task Force became highly focused around three commodities of particular interest: oxytocin, Misoprostal, and magnesium sulphate. 8.2 A Proactive RHSC The emergence and the willingness to embrace the Caucus on Underutilized Methods and Maternal Health Task Force exemplify a Secretariat that is “leading from behind”, as both staff and members describe it. In recent years, however, this approach has been complemented at times by leadership from the uppermost levels of the RHSC. The organization’s maturity combined with its growing stature within FP/RH, and development more broadly, has made this possible. In subsequent revisions and updates of the RHSC Strategy and Terms of Reference, which the evaluators recommend, this emerging role of the RHSC will need to be clearly considered and articulated. Maintaining balance and trust within an organization that is de facto built on consensus among the membership is important. In the following are two instances where the RHSC effectively exercised leadership on behalf of its members and to advance the field. 8.2.1 Manufacturers The convening power and neutrality of the RHSC opened the way to developing relations with the manufacturers of FP/RH commodities. That this has occurred independently from the donors and governments directly involved in procuring these products has had distinct advantages. Encouraged by previous Executive Committee Chairpersons, the RHSC director met with various contraceptive manufacturers. Being better informed about products and perspectives proved mutually beneficial. Though some members initially opposed the idea, eventually several large pharmaceutical companies as well as generic manufacturers involved in FP/RH commodities became RHSC members. To date, their engagement has not been at the level of active Working Group participation. However, the neutral space that the RHSC provides has brought into the open a number of issues. Perhaps most notable among these is that price negotiations occur hand-in-hand with deliberations about supply predictability, volume, and sustainability. 46 Evaluation of the Reproductive Health Supplies Coalition 8.2.2 Hand-to-Hand Campaign The Campaign to focus global attention on meeting unmet needs of 100 million women for contraception by 2015 started as an idea at the highest levels of the RHSC. As related by both the current Chair and the RHSC Director, Hand-to-Hand emerged as an opportunity to focus global attention on an important issue. As such, it has already had the impact of galvanizing donor attention around the upcoming Family Planning Summit to reinvigorate funding for family planning towards a revised metric of 120 million by 2020. Among members, however, it is not clear whether Hand-to-Hand has been a hallmark success or a not-fully- thought-through initiative that will disappear as quickly as it came. The higher level perspective within RHSC is that “Hand to Hand is one where the community comes together and says we will work towards this, we will work towards this goal.” As further elaborated by one of the conceptualizers, there was also strategic sense behind it: “(The) Hand-to-Hand campaign has been a really strong mobilising force for the Coalition and has given it a sense of purpose …(The RHSC does not have) a kind of really strong uniting purpose…Lots of very interesting people, lots of individual commitments to the issues, but nothing that brought us together to deliver something tangible. So that for me was the purpose behind Hand to Hand.” Some advocates within the RHSC however see Hand-to-Hand as something that lacks strategy, not an approach that imparts it. The following extracts from key informant interviews convey this clearly: “(Hand-to-Hand) happened so quickly…there were a lot of questions: Toward what end? What’s the goal here? What are we attempting to achieve? … we’ve discussed within the RMA Group of how do we orient our strategy, our work, in support of Hand-to-Hand, but … it’s not really clear what the Coalition is seeking to achieve…in the RMA Group, where there is interest and willingness, but not certain that this has staying power…misunderstanding of advocacy as PR, as communications…(when) it’s much deeper than that.” “I wasn’t really that involved in it at all, but it just seemed like it was a campaign that had good intentions, … just hired a private firm to basically create a campaign… which isn’t really how we do campaigns because there’s a lot of legwork and a lot of prep work and getting partners on board.” 8.3 Coalition Upsides/Downsides Like any organization, the RHSC has strengths and weaknesses. For example, members with separate funding well aligned with a particular Working Group are often the most active participants, usually the central office staff of large international organizations. In this way, the Systems Strengthening and Resources Mobilization and Awareness Working Groups were historically better endowed than the Market Development Approaches Working Group. In part the Innovation Fund was created to address this imbalance. A related but different imbalance apparent across all Working Groups is the lack of country- level participation and involvement. Even when Working Groups, such as the RMA and MDA, have engaged in country-level activities, lack of country representation continued. The recent election of the country representative, himself a staff member of an international organization separately funded for an advocacy project, to co-chair the RMAWG is a notable exception and worthwhile step in the right direction. 47 Evaluation of the Reproductive Health Supplies Coalition While key informants indicated awareness of such imbalances, members have not as yet been actively engaged to find solutions. This is likely to be a leadership role for the Secretariat to undertake, perhaps in conjunction with revising the Strategic Plan. Some key informants, donors as well as implementing organizations, did express that the Secretariat should exercise more strategic leadership, including taking positions on behalf of the Coalition with regard to particular issues. Instead of providing background support, for example, to the upcoming FP Summit, the RHSC could put forth a position on the Summit itself. This would clearly be a diplomatic and delicate task. At present, means for gathering consensus among members are not clear nor would this likely occur without some dissension among the ranks. The foundational documents, the 2007-2015 Strategic Plan and the 2009 Terms of Reference, firmly fix the RHSC as a members-driven organization. The related mantras have been that the Working Groups are the engines of the Coalition and that the Secretariat leads from behind. However, from the examples above, it is clear that these perspectives are no longer entirely accurate. As the RHSC matures, it will be useful to define and describe its structure and operations in a way that reflects how it does and will likely continue to operate. Bottom-up, member-driven proactivity as well as top-down leadership will both be important for the RHSC to succeed in the coming years. It is important that members understand and embrace both approaches. 48 Evaluation of the Reproductive Health Supplies Coalition In 2004, “No Product, No Program” went mainstream when the Interim Working Group on Reproductive Health Commodity Security reformulated as the Reproductive Health Supplies Coalition. Today, nearly 200 organizations are members of this Coalition. In this section, we assess the readiness and capacity of the RHSC to sustain this movement and continue advances for FP/RH commodity security. 9.1 Country Level: Origin and Destination In recent years, many advances have occurred within contraceptive and reproductive health commodity security, a number directly attributable to the RHSC. However, when one key informant, a well-placed donor representative, considered future pathways for the RHSC, he raised an important question and recalled a seminal study commissioned earlier by the Gates Foundation: “Why don’t we have country-level impact? We still have that study from way back when, Mercer, that showed end-to-end supply chain: where are the problems that ultimately manifest themselves in stock-outs? If you attack the supply chain at country level, you make about, I think they said, 70% difference. If you attack the problem at higher, global level, which is where we focused: about 30% difference. Let’s be honest, that’s what we did – Pledge Guarantee, AccessRH, CARhs. Fine! Done that! Let’s get to the 70% now, and that would require turning the Coalition on its head!” Since the 2005 RHSC Meeting in Seattle, where the Mercer Study as well as country mapping were both presented, country-level involvement has been a recurrent theme for improving FP/RH commodity security. This perspective continues to remain essential and is central within this consideration of RHSC’s potential for continuing to make advances for FP/RH commodity security in the future. Key informants were nearly unanimous about need for the RHSC to focus more on country- level impact. As one former high-level director of a large donor agency stated of the RHSC, “They have to get more involved, and (its Director) readily admits this; they have to get more involved in the South. Contraceptive security is never going to be realized unless countries take more ownership in contraceptives, in their purchase, and their distribution.” However in terms of what should actually be done, key informants were much more divided. Opinions ranged from doing more to keep country pages on the RHSC website current to becoming directly involved with members in supporting country-level activities and interventions. As shown below, a number of respondents independently concluded that the RHSC should focus within a select number of countries: “The Coalition can bring value … and maybe one can redefine (its) role in a way (as) technical assistant/ advisor/consultant… which they could send to countries to help to solve problems (by) consulting, (providing) expertise, knowledge, technical assistance…if you think about supply in a broader sense beside products, you even can come up with supply of 9. Does RHSC seem poised and/or have potential for improving and increasing contraceptive security in the future? 49 Evaluation of the Reproductive Health Supplies Coalition information, supply of advice, supply of consultancy work… select maybe one or two or maybe three pilot countries… (and) get to a more, let’s say, cross-country way of learning.” “I wonder what the Coalition could perhaps have tried to have done to become relevant in a small selection of countries… a strategic opportunity for the Coalition to say we’re going to try and make that happen in these five countries… to demonstrate that the Coalition’s in a position to really make a difference. That the Coalition, that global thing, in these five countries can do that.” “at the country level… you (could) have a country or a region representative from the Secretariat going around specific countries engaging people, meeting with them regularly, you know kind of encouraging their participation, disseminating tools and resources, all of that… if it were a focus country to encourage southern member participation in the Coalition then I could see that being more relevant than the way (the RHSC is) currently set up.” A respondent to the electronic survey of members, a staff member of a private foundation, added this comment: “I am fully supportive of the Coalition being active in 2-4 countries and testing whether they can build a comparative advantage that serves the field”. Another respondent, this time from a bilateral donor agency, was more expansive on the same theme: “The early success of the RHSC has been focused on improved coordination,raising commitment,and the development of innovative supply approaches at the HQ level.There is an opportunity to amplify the impact of the RHSC by strengthening coordination, commitment, and innovation at the country level.This is happening already in some countries, but the RHSC could make these efforts more intentional and expand effective approaches to more countries.” In fairness, not all respondents saw country-level as a present strength or future emphasis for the RHSC. A dialogue within one large INGO between key informants made this point very clearly: Respondent 1: Could (the RHSC) do something that brought more country people onto the same page? Yes. Would it necessarily happen in-country? I don’t think so. I think the Coalition’s power lies outside the country and it may be regional or it may be global… Respondent 2: And I go back to the Contraceptive Security indicators. . Who is on it? By getting that information out in a decipherable way to the Coalition members, the countries, maybe it helps to call attention to what is or what isn’t happening. It is a data point that helps to track and hold them accountable by just sharing that information. But beyond that I don’t see the Coalition doing anything. Respondent 1: Yeah, I don’t see it at the country level. In general, however, most key informants and member survey respondents were favorable to the RHSC being more actively involved at country level. Of 65 who completed the electronic survey, 55 agreed with the statement that “the Coalition should engage more at country level to expand FP and commodity security.” Interestingly, of these only 4 of the 9 respondents from INGOs agreed. 9.2 The RHSC Membership The Coalition has grown to nearly 200 member organizations. Building on an original core of some 15 organizations, the RHSC is currently increasing by an average of 10 new members 50 Evaluation of the Reproductive Health Supplies Coalition a month. Having surpassed the rolls of the IPPF, the RHSC is likely the largest extant body able to call on organizations to join efforts for improving FP/RH. The Director put this expansion in perspective for considering RHSC’s future: “Now when you have almost 180 members (in March 2012) and you are really probably the largest single partnership family planning organization in the world, then in a way it kind of. it demands more of you and there are responsibilities that come from having that. And do you say, ‘Well, that’s fine’ and I ignore that and still keep on commodities, or do you kind of use commodities as a leverage to be able to help you address the broader issues that do touch upon issues important to the wider community? It is a good question and it’s one that we’ve discussed a lot.” Increasingly well regarded and widely recognized, RHSC now includes a wide array of stakeholders. In addition to the core INGOs and donors, the membership includes growing numbers of multinational pharmaceutical companies; generic manufacturers; international organizations involved in related health and development issues; social marketing organizations; government officials; and in-country civil society organizations. Membership however does not necessarily mean active involvement or participation. As a SSWG member observed during its May 2012 meeting, service delivery INGOs were not at the table. While service delivery organizations sometimes participate in the SSWG, those who regularly attend from PSI, MSI, or IPPF/ICON represent the commodity units of those organizations. Others who could notably add to Working Group discussions are civil society organizations and manufacturers. To its credit, the recent MDAWG meeting in Palo Alto did reach out to include some supply industry representatives. Operating in conjunction with the RMA Working Group from 2006 was Project RMA, led by Population Action International and funded by the Gates Foundation. Project RMA was an international advocacy initiative at global, regional, and country level. However, though Project RMA had notable achievements within a few countries, the weight of this Project ultimately proved deleterious to the RMAWG, particularly those members who did not have access to these funds. As some key informants reported, the end of Project RMA led from 2010 onwards to revitalization of the RMA Working Group. A contributing factor has been a similar project, Advance Family Planning (AFP) with funding from several donors. AFP appears to have successfully used the RMAWG as a platform from which to engage civil society organizations to advocate for commodity security in several African countries. If a current request for renewed funding is successful, the AFP and the RMAWG will continue active involvement at country level in coming years. With membership likely to continue increasing, the future of the RHSC depends on three inter-related factors: member commitment and involvement; funding to support member initiatives, including those of the Working Groups, the Secretariat, and its value-added contributions; a Strategic Plan that is re-focused and revised, likely moving towards having impact further down supply chain. 9.3 Breadth and Depth of the RHSC A perennial issue within the RHSC concerns how broad should its substantive reach be. Some members expressed concern that moving beyond contraceptive supplies could be detrimental to RHSC support for FP. However, others have encouraged and embraced this in such areas as maternal health supplies. For example, the Task Force on Maternal Supplies, which emerged from member interest, was not embraced by all. As one donor long associated with the Coalition noted, the RHSC “is broadening out, you know, to do maternal health supplies and to do supplies during emergencies, etc. I think that it’s good that they’re 51 Evaluation of the Reproductive Health Supplies Coalition doing these things, but again (it’s important) to keep everything in balance towards the priority (of contraceptive commodities).” Another donor made a similar observation, but put this into an even broader context: “(The RHSC has) been very flexible in adapting to opportunities that arise and structuring themselves and the members in a way that delivers and meets the needs of new constituency groups and new opportunities. I guess the question for all of us: when new things come along is: what drops off?.Does it…strengthen its broader reproductive health mandate? Does it move into some of the other stuff around demand, crystallization around operational planning and (service) delivery?” Some key informants expressed concern that moving beyond contraceptive supplies could be detrimental to RHSC support for FP. However, others have encouraged and embraced broader involvement in such areas as maternal health supplies. As one long-time member of the MDA Working Group reflected, “It is all procurement. And as contraceptive prevalence goes up and governments will move into a range of areas, there is no question that we have to think about (other commodities) this as an entire market.” Responses by key informants to broader extension – beyond commodities to advocacy and even service delivery – were more mixed. The RMA Working Group, for example, supports some country-level advocacy, with technical and financial support from Advance Family Planning promoting, government commitments to and procurements of contraceptives. Though some key informants argued that demand generation was also important, for many, including donors, this would be beyond the purview of RMAWG and AFP as well as the RHSC itself. Similarly, for service delivery, though having service provision organizations in the Coalition is important, extending RHSC in that area would likely be limited. The breadth of RHSC can be reconsidered within the update of its Strategic Plan, However, with form following function corresponding changes may also be needed for the Secretariat, adding staff for example who have policy and possibly service delivery expertise. Most key informants wanted to maintain a core focus on commodities; however extending to areas beyond contraceptives was widely if not totally accepted by all. Similarly, member survey respondents who favored greater RHSC involvement at country level were evenly distributed among the areas where they thought the RHSC should engage. Selecting among three of seven choices, most respondents (55%) opted for resource mobilization and financing of RH commodities. However, equal numbers (49%) also selected increasing demand for FP/RH services/supplies and supply chain strengthening. For some interviewed, the concern that started more than 10 years ago to get the field more fully focused on commodities has now come full circle. As one of the originators of this movement stated: “Social change is not going to happen by putting out a bunch of contraceptives only… in the past 20 years people have shied away from that aspect of what we always used to called IEC and what was always part of the family planning programming was changing people’s minds – one woman, one couple, one community, one district and one nation –and nobody does that anymore.” This same topic was also raised during the recent SSWG discussion on future plans and directions. In the process, the relative lack of service delivery organizations among RHSC members was noted. 52 Evaluation of the Reproductive Health Supplies Coalition Since 2006, the RHSC has maintained a structure of Working Groups within which members work to advance FP/RH commodity security. The RHSC website provides a platform to communicate not only the results of that work but also broader developments within FP/RH commodity security. To maintain the website, the RHSC relies on inputs from members and outsources technical updates to consultants. Though some members remarked that the website lacked depth and was in many areas not current, it remains a good repository of what has occurred within the RHSC structure. As a portal to resources, data bases, and logistic system learning opportunities, the website is well positioned and appears well used. The Secretariat, again with members’ input, also produces a monthly newsletter, a useful update on relevant achievements and events occurring across the field. Clearly with resources and attention, the website may be improved. A solid base to build upon, the website could also considerably expand. More attention could be given, for example, to making the website a more interactive forum for information exchange. The development of Learning and Professional Training Opportunities (LAPTOP), a tool to connect supply chain managers with web-based courses on procurement, logistics, contracts, and other related operations, is a step in the right direction. Similarly, the Secretariat could use the website to bring focus to issues of particular importance for FP/RH commodity security. As one example, the RHSC could monitor funds pledged during the FP Summit. Posting expenditure information on the RHSC website would help turn this July event from the metaphorical wedding into the lasting marriage that some donors hope for. Similarly, should the RHSC place priority on outcomes within some countries, these could be highlighted within revitalized attention to selected country pages. 9.4 Poised but Not Fully Positioned As its rapidly growing membership indicates, the Coalition is clearly poised to improve and increase contraceptive security, both as a field of knowledge and through practical applications to improve systems at various levels. However, it does not currently have the bandwidth or depth among Secretariat staff to do so, nor does its dated strategy provide specific pathways for moving further along the supply chain. A woman in Ethiopia who receives Implanon becomes an ultimate beneficiary of price reductions that the Coalition helped to broker. However, such direct line connections between an RHSC contribution and a result remain rare. The challenge for the future, and one that was embraced by the majority of key informants, is: How can the RHSC position itself to have greater impact in the coming years? Already being discussed among members, the evaluators suggest that revisiting and revising the RHSC strategy would usefully concentrate attention upon this important question. The ethos of being member-led can and could be balanced by a Strategic Plan that orients Working Groups towards promising lines for advancing FP/RH commodity security. Instead of the current RHSC Strategic Plan that encompasses and coordinates within what member organizations already do, the expanded member-base itself could be involved in revising the strategy towards higher level goals of what can be collectively accomplished. In responding to the next and last question (Section 10), the evaluators highlight some additional ideas and recommendations for future consideration that emerged during this evaluation. 53 Evaluation of the Reproductive Health Supplies Coalition 10.1 Strategic ways forward 10.1.1 Retain core focus on RH commodities There is clearly some pressure within the Coalition to widen the focus of commodity security. The evaluation team recommends retaining the current focus on RH. Currently this includes a comprehensive array of family planning products as well as a more focused set of maternal health supplies. Though the Coalition has contributed to repositioning the supplies agenda, there is still a long way to go. Contraceptive commodity security achievements in particular could well be compromised without continued Coalition attention to family planning. However, encouraging coordinated action to secure a slightly wider range of related health supplies may in time become politically practical as well as practically achievable. 10.1.2 Maintain global as well as local focus on supply chain improvement The question of whether the Coalition should operate more directly at country level was much discussed in the course of this evaluation. Many Coalition members recognize that commodity security remains weak in many countries and encourage greater RHSC involvement and efforts. The Coalition accepts this as a shared objective requiring greater collaboration to achieve. Key informants proposed various options for RHSC engagement further along the supply chain, from supporting mini-coalitions in some countries to taking concerted action in a small number of selected focus countries. However, beyond broad agreement for doing more to engage at country level, there was no consensus. The evaluation team recommends that the RHSC maintain its global-level focus, continues to reach toward countries through existing regional forums, and considers concentrated engagement in a select number of countries where value would be added. 10.1.3 Promote greater country level ownership of the Coalition The evaluation team also recommends increasing country-level ownership of the RHSC by making the Coalition more relevant to country needs. The Strategic Plan states: “While international expertise can play a critical role in recognizing weaknesses and identifying technical remedies, the search for long-term solutions will only succeed when countries themselves define the roles of their partners, including the Coalition, and set their own supply agenda. The Coalition will, therefore, reinforce the principles of country leadership and 10. Beyond serving as an umbrella for collaboration, what alternative models or approaches could and should the RHSC adopt that are needed in the coming years to advance RH commodity security globally, regionally, and at country level? 54 Evaluation of the Reproductive Health Supplies Coalition ownership in all country-level RH supply security activities”. This remains relevant for the work of the Coalition but in practice this is not how the RHSC operates. The Coalition should consider what it can do at global level to help countries. Could country- level members identify common critical issues in commodity security to which the Coalition can respond? Does the RHSC simply need to ask its country level members: ”What could we do to help you?” This would not be a straightforward process. At the moment the voices of southern partners are not often heard within the Coalition. With the exception of one co-chair, Global South engagement with the Working Groups is virtually non-existent. As currently structured, this means they miss out on much of the agenda setting. The team experienced difficulties involving Global South members in this evaluation, not unlike those that the Coalition faces on a regular basis (e.g. poor phone lines, lack of internet connectivity). However, like Coalition members generally, country-level contributors to this evaluation, both key informants and member survey respondents, clearly expressed value in being members. Some also provided useful input on why Global South engagement is limited. Cost is a big issue, not only to attend meetings, but even joining conference calls is constrained by cost as well as time. Also limited is knowledge of RHSC tools and approaches and their potential benefits for those who would immediately benefit, such as Ministries of Health. Strategies to increase the participation of Global South members could include: � Developing different forms of engagement with the Working Groups, which currently rely on members attending and actively contributing. For example, members could undertake consultative exercises about new priorities, through email or by reaching out through their local country offices. � Having one or two country members on the Executive Committee in revolving posts. � Forming a country Working Group which, borrowing from the Caucus practice, informs the work of other WGs. � Funding participation by southern partners in Working Group meetings (i.e. paying for phone calls, selected travel). Ideally, each WG would have a budget to cover such expenses on a regular basis. � Funding attendance by more strategically selected southern partners at the Annual Meetings and working out how to better engage them, and subsequently sustain that engagement. � Developing communication approaches to reach existing ‘silent’ southern members. � Developing mechanisms that encourage/promote cross-country exchanges and interactions on commodities to create country level momentum. � Supporting more links between northern and southern organizations, as has occurred in some countries through the RMA Working Group. This could be applied across the RHSC more broadly. 10.1.4 Retain but supplement the partner-centric funding model The Strategic Plan states that ‘the Coalition relies on the contributions and mutual accountability of its members.’ In 2007, the Gates Foundation was the only donor providing core support. Some additional funding was subsequently made available by BMGF and managed through the Innovation Fund. At a total of $2m this was enough to initiate ideas, but not enough to ensure longevity. Realistically the RHSC will remain largely dependent on the contribution of members, not a bad arrangement in many ways. Volunteerism increases members’ stake in making the RHSC work, focuses the Secretariat on the need to provide added value to members, and encourages everyone to prioritize. It is also a good barometer 55 Evaluation of the Reproductive Health Supplies Coalition of how well the Coalition is doing: volunteers can and do walk away if they do not get the returns they want. Despite such advantages, some Working Groups have felt disadvantaged. At different times and to varying extents, RMAWG and the MDAWG members have not had flexible funding to support Coalition activities. The evaluation team recommends that consideration be given to providing selective financial support to Working Group functions and operations. Different from Innovation Fund support, this suggested supplementary funding would support Working Groups to function effectively. It could be allocated as budgets for each Working Group or controlled centrally on a case by case basis. Ensuring this did not act as a disincentive for members to invest would be important. Such supplementary funds could also be usefully tapped to facilitate and engage southern member participation. The other problem with lack of funding to the Coalition is that some initiatives wither before being fully and fairly tested. The PGH is a case in point. The evaluation team recommends that consideration be given to housing some of the best but most vulnerable initiatives in the Secretariat, starting with the PGH. The PGH has the added advantage that it could potentially be self-financing and eventually generate funds which could perhaps be used to support other initiatives. The preparedness of donors to use PATH as a pass-through arrangement, and the fact that this arrangement would only apply to home grown initiatives may remove some of the potential problems identified below with ‘ready formed’ initiatives. The evaluation team also recommend that the Innovation Fund be renewed, renamed, and reoriented. The Innovation Fund supported some useful work, has helped pull some Coalition members together, and generated momentum for a few Coalition-germinated initiatives. However, Innovation Fund grants were won by agencies which not only put forth good ideas but which were also good at developing proposals. As a result, small southern NGOs, inexperienced in producing international quality grant applications, were not successful despite best efforts to include them. Also many of the ideas funded by the Innovation Fund were not actually very innovative, even though they were useful to have done. A renewed and expanded funding stream should be instituted to encompass a wider range of initiatives, tailored to different constituencies of the Coalition. This would place priority on collaborative projects and activities and could include: � Innovation Fund for truly new ideas � Dissemination / Continuation Fund for in-country field testing and further development of existing tools � Country Catalyst Fund to support advocacy and other initiatives, e.g., provide support around upcoming events, important country-level decision points, or potential supply system or service delivery breakthroughs. This Fund would particularly target country- level activities and organisations. Building on support that PAI is currently providing to local NGOs, including similar upfront support for proposal development would also be useful . 10.1.5 Step up to the RHSC role as a leader in the field There is potential for the Coalition to take a stronger role in leading on RH commodity security. That it has many and possibly all the key members for commodity security at global level gives RHSC a potential voice and power not fully exploited to date. Caucus efforts to have FP commodities included by the UN Commission on Life Saving Commodities for Women and Children, though a response to an omission, is an example of what can be done. There may be potential for the Coalition as a body to be more proactive in identifying key issues, taking a clear public position, and moving the agenda forward. The HandtoHand 56 Evaluation of the Reproductive Health Supplies Coalition Campaign had the potential to do this but has been overtaken by the Family Planning Summit. In the future, a stronger leadership role for the Director may be useful. At present, the Director is essentially the lynchpin of the membership network, the Coalition’s ‘front man’, someone who moves things along, often leading from the middle rather than the front. This role was appropriate in the earlier days of the network when the key task was to bring together a disparate, sometimes cantankerous membership; it remains valuable for maintaining neutrality. But there may now be scope for being more proactive on some strategic issues. This may imply more of a mandated role, with the annual meetings used to build member consensus around issues and set strategic direction. 10.1.6 Create a closer relationship with UNFPA’s GPRHCS UNFPA, with its Global Programme to Enhance Reproductive Health Commodity Security (GPRHSC) is a key member of the Coalition. The Programme is unique for many reasons not least of all its scale, country reach, emphasis on both commodity supply and capacity building, flexibility, and ambitious goals very much in line with what the RHSC is trying to achieve. The Programme is now going through a process of revision for new donor funding. The close relationship between the Coalition and UNFPA has been important for both parties, and UNFPA staff are active within RHSC operations in different ways, e.g. Executive Committee, Working Groups, some of the Innovation Fund initiatives, and in the delivery of AccessRH. However relations between Coalition members and Programme implementers have been more tenuous. As a result they do not always work together at country level in the most effective way. The evaluation team recommends that the RHSC and UNFPA give special attention to how the GPRHSC and members programmes can be mutually enhanced, both at global and country level. This should involve the RHSC having a more formal link with the Programme (perhaps a seat on the proposed consultative body) and some strategizing on how lower level links can be created and operations of the GPRHSC enhanced. 10.1.7 Continue to encourage regional forums, with provisos Encompassing regional forums into the Coalition is both pragmatic and promising. Placement of a regional advisor under the Hewlett grant to SECONAF is an opportunity to influence developments in that region and learn from that experience. The SECONAF experience should be tracked for lessons learned for having impact at country level. It is not yet clear however whether regional approaches will generate country level impact; moreover no metrics currently exist to monitor this for LAC, an omission in the MEF which should be addressed if politically possible (and certainly should be addressed for SECONAF). The evaluation team recommends that the Coalition continue support for the two regional forums. However, the Coalition should remain cautious about regional approaches generally, for which there are many more failures than successes. Groupings of countries (like-minded, same stage, or self-selected to learn from each other) may be a better way to proceed and this could be encouraged within the regional forums. 10.1.8 Revisit the Strategic Plan As discussed previously, the current Strategy (2007 – 2015) has served the Coalition well but is now somewhat out-of-date. The evaluators recommend that the Strategic Plan be refreshed and revised to reflect the expanded membership and in response to the suggestions above. An explicit theory of change would usefully contribute to both formulating an updated strategy and revisiting the MEF. 57 Evaluation of the Reproductive Health Supplies Coalition 10.1.9 The comparative advantage of the RHSC should be preserved The strengths of the Coalition, as discussed in section 4.1, are its neutrality, the breadth and depth of its membership, its power as a convenor, the commitment of its members, and the effectiveness of its Secretariat. RHSC neutrality has been hard won and is its greatest asset. It is essential that this and its other strengths are preserved even as the RH landscape, in general and for commodity security in particular, evolves and new potential opportunities for the Coalition emerge. 10.2 Operational issues 10.2.1 Secretariat The evaluation team recommends that the Secretariat has its capacity increased, to both better service the existing needs of the Coalition and step up its leadership, as discussed above. The Director needs more time to lead more pro-actively. Increased technical capacity within the Secretariat would free the Director for higher level leadership and provide valuable support to the Coalition structures. Secretariat capacity is currently increasing with the addition of full time communications and M&E officers. The evaluators suggest that RHSC capacity needs be further considered as the landscape for RH generally and RH commodity security in particular continues to develop. 10.2.2 Executive Committee The evaluation team recommends that the Executive Committee be reconfigured to permanently represent more member constituencies (e.g. manufacturers and country-level members, if practicable), and to bring on some independent strategic thinkers to act as short term advisors, with terms of perhaps 1 to 2 years. This would enable the Executive Committee to widen participation, and to be continuously challenged. 10.2.3 Annual meetings, convenings, and networking among members Key informants both value and criticize RHSC Annual Meetings and other Coalition gatherings such as Working Group meetings. The evaluation team recommends that RHSC continue to conduct annual meetings and support other forums where members come together, but with some needed improvements. Annual meetings, for example, could be a useful forum to formally strategize within the Coalition as a whole. Similarly, these occasions could be used to build consensus among members around issues on which the RHSC would take leadership. Annual as well as other meetings also need to be used to engage countries more systematically. Expanded support for strategically selected participants as well as for follow up would provide added value. Holding annual meetings within strategically selected countries, as has occurred in Uganda and Ethiopia, should continue. These have proven useful venues to mobilise government support, show-case successes, and encourage members to lend support to future efforts. The RHSC needs to continue being strategic about where and with which members to undertake such investments. Given the tremendous need for FP/RH in francophone Africa countries, the October 2012 meeting in France seems another strategic venue selection. On a practical note, the Secretariat needs to upgrade how it organizes and facilitates its Annual Meetings as well as other meetings convened under the aegis of the RHSC. The evaluators suggest that the Secretariat identify a core group of highly skilled facilitators to plan and conduct such meetings. Methodologies, such as Open Space used with the 58 Evaluation of the Reproductive Health Supplies Coalition SECONAF contingent as part of the Addis annual meeting, should be expanded to include other new approaches. For example, Future Search and World Café have been successfully applied in a wide variety of settings to fully engage even very large groups, More importantly, by consulting with professional facilitators the RHSC will conduct meetings that are both more vibrant and efficient. Complaints from key informants that meetings were often long, boring, or inefficient will disappear. Beyond meetings, there is a burgeoning field of practice around working with large networks that the RHSC has yet to tap. Social network analysis is the academic appellation for understanding and tracking the power of individuals and groups working towards social change. Beyond commodity security, Coalition members are fundamentally committed to improving equity and increasing opportunity. Learning from others similarly committed, though differently focused, will help the Coalition more fully harness as well as harmonize what members working together towards common ends can achieve. 59 Evaluation of the Reproductive Health Supplies Coalition Evaluation of the Reproductive Health Supplies Coalition Background The Reproductive Health Supplies Coalition (RHSC) was established in 2004 as a small, loosely organized group of 12 organizations, mostly donors from the public and private sectors, with a shared dedication to making reproductive health (RH) supplies available to people in developing countries. The RHSC was born out of the need for better coordination and collaboration among these organizations in such areas as global advocacy, resource mobilization, and supply chain strengthening for RH commodity security. From early on, though, it was clear that for the RHSC to be effective, its membership needed to grow and embrace a wider range of stakeholders, including to be more representative of the “total market” for family planning(FP)/RH supplies. The RHSC has thus grown to now include more than 150 member organizations, including developing country governments, international and country-level non-governmental organizations, civil society organizations, technical agencies, bilateral donors, multilateral organizations, private foundations, regional bodies, and manufacturers. It is the only global partnership of its kind for reproductive health and family planning. The present RHSC structure is comprised of a Chair and Executive Committee, a Secretariat based in Brussels (including several full-time staff and an Executive Director), three technical working groups (resource mobilization and awareness, market develop approaches, systems strengthening), two regional fora (Latin America/Caribbean, Francophone Africa), and one caucus (new and underused RH technologies). The RHSC has Terms of Reference (last updated 2009) and a Strategic Plan and Monitoring and Evaluation Framework for 2007-15. (These documents, as well as working group work plans and other background materials are available at www.rhsupplies.org). As detailed in the Strategic Plan, the RHSC’s vision is for: “all people in lower- and middle income countries to be able to access and use affordable, high-quality supplies, including a broad choice of contraceptives, to ensure their better reproductive health.” The RHSC’s 2007-15 Strategic Plan encompasses three Strategic Goals, which in turn encompass 25 objectives: • Goal 1: Increase the availability, predictability, and sustainability of financing for RH supplies. • Goal 2: Strengthen the capacity of health systems to deliver RH supplies in a sustainable manner. • Goal 3: Assure the added value of the Coalition as a productive and sustainable global partnership through support for efficiency, advocacy, and innovation. To realize its goals and objectives the RHSC and its members employ four broad operational strategies – collaboration, advocacy, innovation, technical solutions. These are further detailed at http://www.rhsupplies.org/about-us/what-we-do.html. The Coalition has an annual membership meeting, while the Executive Committee, working groups, regional fora, and caucus meet more often, whether on a set frequency or as needed. Apart from the limited case of the RHSC Innovation Fund, the Coalition is not a funding organization per se, like such other global health partnerships as the Global Fund for AIDs, Tuberculosis, and Malaria, or Vaccine Alliance. Rather, it works in a very partner-centric way, where activities in support of the working group work plans are largely funded and Annex 1 Statement of Work 60 Evaluation of the Reproductive Health Supplies Coalition implemented by RHSC members. (The Innovation Fund provides only limited grants to members as “seed funding” for activities that are aligned with working group work plans.) There is dedicated operational funding, currently provided by USAID, UNFPA, DFID, and the Gates Foundation, for the RHSC Secretariat, which itself has a budget and work plan approved by the Executive Committee. This funding, since 2010, has averaged approximately $2 million per year, and the current phase of funding will end in 2012. The Secretariat will develop a new 2013-2016 work plan and budget for presentation to the Executive Committee, and this evaluation is being commissioned to inform the process, as well as inform decision making by potential donors for the next phase of RHSC funding. This is the first such evaluation of the RHSC. Assessments of narrower scope have been conducted, including a review of Coalition governance policies (2007), sustainability review (2008), survey of Coalition members on the role of the RHSC Secretariat (2008), member- wide survey of the Coalition’s Strategic Plan (2011), internal review of the Innovation Fund (2011), and a review of the Coalition’s communications strategies and tools (2011). Statement of Work This evaluation is both a performance evaluation as well as an enquiry into the optimal structure and function of the RHSC and its components. As such, it has three overall objectives: • Achievement: Assess how well the RHSC has met its objectives and made an impact on RH commodity security. • Structures/Processes: Assess how well RHSC structures and processes have worked towards achievement of RHSC objectives. • Future Directions: Recommend how the RHSC’s role should evolve in light of the current and emerging environment, and identify measures to improve the efficiency and effectiveness of the RHSC. Key questions to be addressed by the evaluation team are listed below. The following list is not meant to be comprehensive, but serves as a guide for interview questionnaires and other data gathering in the evaluation. While the focus of this evaluation should be the Coalition since 2009, it will need to delve into earlier history in order to address some questions, particularly those as they pertain to Coalition achievements and the role of the Secretariat. A. Achievement 1. To what extent has the RHSC fostered progress for RH commodity security at all levels (global, regional, country), and, to the extent that it can be gauged, what has been the impact of the RHSC on increasing access to quality, affordable RH supplies for clients in low and middle income countries? • What progress has the RHSC made specifically against its Strategic Plan, as measured by the Monitoring and Evaluation Framework? • How successful has the RHSC been in increasing the prominence of RH commodity security in global health and, more broadly, international development efforts? • How successful have RHSC-inspired initiatives been in terms of country uptake and results? 2. Does the RHSC have and has it worked according to a demonstrably effective “results chain” or theory of change, where the Coalition’s operational strategies (collaboration, advocacy, innovation, technical solutions) have guided inputs that in turn have translated into outputs and outcomes for RH commodity security? 3. What is the RHSC’s comparative advantage? What added value does the RHSC bring to the individual efforts of its members, and more generally to international, regional, and country-level efforts for RH commodity security – over and above what would be accomplished without the RHSC? 61 Evaluation of the Reproductive Health Supplies Coalition B. Structures/Processes 1. To what extent, and to the extent that it can be gauged, does the RHSC achieve value for money in terms of activities undertaken and results achieved? Have RHSC- managed resources/assets been optimally managed? 2. How able or nimble is the RHSC to pursue new opportunities and accommodate new ideas and new areas of focus (as in the work, for example, of the Secretariat and Working Groups)? 3. To what extent are RHSC structures and processes appropriate for its mission? How well has the RHSC worked to ensure efficient and effective working by its working mechanisms, individually and collectively – considering their respective roles, structures, processes, modalities of working, and working relationships to each other? • How effectively have the Working Groups, Regional Fora, and Caucuses “connected” their work plans to each other, seeking and achieving synergies in work plan objectives, priorities, and activities? 4. How effective have strategic and work plan decision making approaches used by the RHSC and its components been (Executive Committee, Secretariat, Working Groups, etc.)? To what extent are decisions – for example, what is in the RHSC strategic plan and working group work plans – driven by country needs and priorities versus global agendas? How well are constituencies (e.g., developing countries, civil society, private industry) represented in decision-making structures? 5. How effective is the partner-centric model for the RHSC? How effective has the RHSC been at catalyzing partner investments and leveraging partner resources (funds, people, etc.) not only for RHSC-inspired initiatives or the RHSC itself, but for RH commodity security in general? • In particular, to what extent has the RHSC’s partner-centric approach to the Working Groups, Regional Fora, and caucuses, and implementation of their work plans, been effective (e.g., dependence on partners for working group coordination and funding of work plan activities)? 6. Are there adequate systems in place to monitor and report RHSC performance? 7. How has the Secretariat’s role evolved over time? Is there evidence that it has assumed greater responsibility for achieving RHSC objectives? What are the Secretariat’s most essential contributions? To what extent is the Secretariat a support to, versus substitute for, the work of other RHSC working mechanisms and RHSC partners? 7 Has the Secretariat enabled the RHSC to accomplish more than it could without the Secretariat? How is it decided whether an activity is the responsibility of the Secretariat or other RHSC working mechanisms or RHSC partners? 8. How effective has the Innovation Fund been in supporting working group work plans, catalyzing significant work that would not otherwise have happened and fostering innovation? 9. With specific reference to operations of the RHSC Secretariat: • How effective has internal management of the Secretariat been? For example, managing human resources and the relationship with PATH. _________________________ 7 The working mechanisms of the RHSC are the Chair and Executive Committee, Executive Director and Secretariat, Working Groups, Regional Fora, and Caucuses. 62 Evaluation of the Reproductive Health Supplies Coalition • Is the Secretariat staffing of the right size and skills mix? Has the Secretariat been able to recruit and retain the kinds of people needed? If not, why? • Are there appropriate controls and oversight for Secretariat finances, and are they exercised appropriately by the appropriate entities? • What have been the impacts, positive or negative, of the decision that the RHSC not be a legal entity and that it is in effect embedded within another organizational structure (PATH’s)? What are the costs and benefits of this lack of autonomy? • Is the Secretariat’s location in Brussels the right balance between costs and benefits, relative to other possible locations, including in a developing country? • The RHSC Secretariat outsources certain core functions (e.g., website maintenance) and ad hoc studies/assessments (e.g., communications assessment). How efficient and effective has this approach been? 10. How effective has the Executive Committee been as a governing body? Is it involvement in Coalition affairs of the right kind and at the right level? 11. Have the Coalition’s relationships with funders of the RHSC worked to best advantage? For example, have disbursements been timely and funding agreements not overly dictating in how funds can used or burdensome in their reporting requirements? C. Future Directions 1. In light of the current and emerging environment, are the RHSC’s strategic objectives and operational strategies the right ones? Is the Monitoring and Evaluation Framework measuring the right things? What changes are needed, if any, to these and to the Strategic Plan to maintain and strengthen the RHSC’s relevance and comparative advantage/value-added, and expand its impact at all levels? • More specifically, what areas of activity should be considered for addition to Secretariat and/or working group work plans, in order to support and monitor countries to strengthen supply chains, build capacity for advocacy, and other that will further RH commodity security? Are there additional working groups to recommend? 2. How can the RHSC broaden the numbers and kinds of organizations, particularly from the global south, that are involved in working group work plans? For example, how can the Innovation Fund, or a mechanism like it, be used to this end? Are there are other means that might be pursued? 3. What measures could improve the performance and efficiency of the RHSC’s components and how they relate to each other, and of RHSC processes (e.g., communication pathways within the RHSC structure, decision making by the Executive Committee)? 4. The Working Groups, Regional fora, and caucuses, while thought of as the technical engines of the RHSC, are particularly dependent on partner contributions, both in- kind and financial. To what extent does this partner-centric model remain relevant/feasible? Should a different approach be taken to supporting these groups and their work plans that would be more efficient and effective, and yield greater impact? What would be the funding implications of an alternative approach? 5. As demands grow on the RHSC Secretariat (for example, to support the HandtoHand campaign), to what extent should outsourcing be expanded versus an expanded in- house staffing? What should be the optimal number and type of Secretariat staff? 6. Considering Questions 7 and 9 in the previous section, what other changes in the Secretariat should be considered to improve the value-for-money of the RHSC? 63 Evaluation of the Reproductive Health Supplies Coalition 7. To what extent are the findings and recommendations of the 2008 sustainability study still relevant? How has the RHSC progressed, however incrementally, towards institutional and financial sustainability, and how can it further its progress over the next 3-5 years? • Considering the answers to Question 9 under “Structures/Processes,” what additional steps might the RHSC feasibly take to further strengthen its human and institutional capacities? • How can the RHSC improve the predictability of its financing? What are the pros and cons to further diversifying the RHSC’s funding, in particular beyond donor funding? Methods A two-person team will conduct the evaluation. The evaluation team will use a variety of methods for collecting, analyzing, and synthesizing data and information. The following essential elements will be included in the methodology, as well as any additional methods proposed by the team. The evaluation team will hold an initial one-day team planning meeting. As part of this meeting, the team will meet with the current donors to the RHSC (the “donor group”), in order to review the purpose and scope of the evaluation, and finalize the evaluation questions, methods, deliverables, and timeline. The outcome of the team planning meeting will be a detailed work plan for the evaluation. In addition to formal briefing and debriefing meetings, there will be scheduled “check-in” meetings with the evaluation team and members of the “donor group.” The evaluation team will provide updates on their progress and, as necessary, obtain additional guidance, data and information. Throughout, the primary point of contact for the “donor group” will be Alan Bornbusch, USAID/Washington. The evaluation team will review pertinent documents, reports, and other materials. These will include, but are not necessarily limited to, the following RHSC documents/materials: • Terms of Reference • 2007-2015 Strategic Plan • 2007-2015 Monitoring and Evaluation Framework and 2009 Monitoring Report • Work plans for the working groups and RHSC Secretariat • Annual reports to donors • Past issues of SupplyInsider, the RHSC’s newsletter • Materials pertinent to the regional fora and caucus • Assessments/reviews commissioned by the Secretariat – Innovation Fund review, communications review, sustainability review, membership survey, etc. • Select RHSC-initiated technical reports/materials – Advocacy Guide and Toolkit, 2009 Donor Gap Analysis, Market Segmentation Primer, Supply Chain Software Guide • Documentation for select RHSC-inspired initiatives such as AccessRH and Pledge Guarantee for Health • RHSC website (www.rhsupplies.org), including the Supplies Information Database • Documentation for a sample of Innovation Fund-supported activities • Documentation pertinent to meetings of the RHSC membership and Executive Committee The evaluation team will conduct qualitative, in-depth interviews with key stakeholders and partners. Wherever possible, the evaluation team should conduct in-person interviews with informants. When it is not possible to meet face-to-face with stakeholders, telephone interviews should be conducted. The evaluation team will develop appropriate interview questionnaires, tailored for different stakeholders. The questionnaires, and list of stakeholders to be interviewed, will be finalized in consultation with the “donor group.” Key informants will include, but not be limited, to: 64 Evaluation of the Reproductive Health Supplies Coalition • RHSC Chair, Director, members of the Executive Committee (the last includes the working group leaders); including, to the extent necessary to address the evaluation questions, past Chairs, Executive Committee members, and working group leaders • Coordinators of the RHSC regional fora and caucus • Representatives of RHSC member organizations, including PATH (given its role in Secretariat management and operations) and others selected to sample: the working groups, regional fora, and caucus; and key RHSC constituencies. The last should include low and middle income country governments, bilateral donors, multilateral organizations, regional organizations (intergovernmental and private), global health partnerships, private foundations, international NGOs, smaller NGOs from the global south and north, civil society organizations, technical assistance agencies, manufacturers. Some oversampling of organizations from the global south will be expected. Approximately 45 interviews will be required. Travel – domestic within the US and international to Europe – may be included to conduct in-person interviews where feasible and warranted. Logistical support for travel will be provided. Deliverables During the first week from the evaluation start, the evaluation team will conduct a team planning meeting8, which will include a meeting with the “donor group” to discuss the scope of work, and finalize the evaluation questions, methods, deliverables, and timeline. The outcome of the team planning meeting will be an approved work plan for the evaluation. The work plan will include, but not be limited to, a timeline for key activities, due dates for deliverables, and schedules for key informant interviews, “check-in” meetings, and debriefing meetings. Prior to submission of a draft evaluation report to the “donor group,” the evaluation team will hold an informal debriefing meeting with the group to present the major findings and recommendations of the evaluation. The draft report will include all the components of the final evaluation report. The “donor group” will provide comments on the draft report to the evaluation team within ten working days of receiving the report. The next draft of the evaluation report will be submitted to the RHSC Secretariat for reaction and comment. An informal briefing may also be held for the Secretariat. A final debriefing will be held for the RHSC Executive Committee, at its spring meeting (expected in late May / June). The final evaluation report will then be submitted to the “donor group.” The final evaluation report should include, at a minimum, the following: executive summary (not to exceed five pages); scope and methodology; major findings; and conclusions of the evaluation team based on the findings, including recommendations for future directions. The report will include as appendices the SOW for the evaluation, list of people interviewed, questionnaires used, and list of documents reviewed. Excluding the executive summary and appendices, the final report will not exceed 45 pages. While the full report is not expected to be a public document, the Executive Summary will be. Duration and Timeline This evaluation is anticipated to commence in early March 2010. It is anticipated that the period of performance of this evaluation will be approximately 60 working days, from commencement of the evaluation to submission of the final report. (Total LOE is estimated at _________________________ 8 This and other meetings with the “donor group” will likely be by conference call. 65 Evaluation of the Reproductive Health Supplies Coalition 120 person days.). The final report is required by late May / early June. The evaluation team will finalize a schedule and exact dates at the initial meeting with the “donor group.” Team Composition A two member team is proposed for this evaluation. Each team member should have 10-15 years of experience in international public health and/or organizational development. The team should have demonstrated experience in the following areas: • Commodity security, preferably for family planning and reproductive health • Organizational development and management • Family planning and reproductive health • Monitoring and evaluation • Design, management, and evaluation of international partnerships, preferably in global health 66 Evaluation of the Reproductive Health Supplies Coalition ACPD final narrative report November 2011 ACPD Interim Report February 2010 ACPD Report January 2011 ACPD RH Advocacy Interim narrative report APA. Organizational effectiveness phase I Report to members Concept Foundation (2009). Accessing Quality Assured Supplies Concept Foundation (June, 2011). Accessing Quality Assured Supplies (AQAS) An MDAWG Initiative Coordinated Assistance for Reproductive Health Supplies Group Progress Report: October 2010 – September 2011 Druce, Nel (7th August 2008). RHSC Membership Consultation, July 2008 Druce, Nel (August 2008). Analysis of responses Eisele, Florian. Reproductive Health Supplies Coalition Defining and reviewing the Secretariat’s communications Equilbres et Populations (2010). EP Interim Narrative Report Equilbres et Populations, (2011) Final narrative report Ethiopia Implant Transaction Press Release Gardiner, Elizabeth, Schwanenflugel, Ditlev & Grace, Cheri (2006). Market Development Approaches Scoping Report Johnson, David, (2007) Report of the Membership Task Force of the Reproductive Health Supplies Coalition JSI Survey Report Manufacturer Input to AccessRH Design JSI (August 2010). Supply Chain Models and Considerations for Community-Based Distribution Programs: A Program Manager’s Guide JSI / DELIVER(2006). Synthesis of Field Advisor Observations on Minimum Volume Guarantee and Pledge Guarantee, Oct 2006 Kinzett, Steve (January 2011). Member Survey on the Strategic Plan Mercer (2005). Contraceptive Availability Study: Methodology and Key Findings Report to the Reproductive Health Field MSI (2009): Final Report ‘RH supplies in Emergency settings’ MSI (2009): Mid Term Report ‘RH supplies in Emergency settings PGH. A New Platform to Improve Efficiency and Impact of Donor Financing for Health Commodities PGH. Information Sheet Population Action International (2011). The Road from Istanbul to Addis and Beyond RHCS: Final RH Financial Mechanism Analysis Business Plan July 2006 RHSC (2008). Annual Progress Report 2008 Annex 2 Bibliography 67 Evaluation of the Reproductive Health Supplies Coalition RHSC (2009) Year 4 Financial Report RHSC (2009). Annual Progress Report 2009 RHSC (2010). Access for All RHSC (2010). Annual Progress Report 2010 RHSC (2010). Secretariat Operations: Financial Summary for 2009/10 RHSC (2010). The Procurement Planning and Monitoring Report: Reducing Contraceptive Stockouts Through Data and Partnerships RHSC (2010). Year 5 Financial Report RHSC (2011). Annual Progress Report 2011 RHSC (2011). Reproductive Health Commodity Security: Leading from behind to forge a global movement RHSC (2011). RMA Working Group: Global and National Advocacy Strategy RHSC (2011). Secretariat Operations: Financial Summary RHSC (2011). Year 6 Financial Report RHSC (April 2008). Financial sustainability planning for the Reproductive RHSC Frequently Asked Questions: Caucus on New and Underused Reproductive Health Technologies RHSC Innovation Fund II Strategy, September 2011 RHSC Innovation Fund Lessons Learned, February 2011 RHSC. Market Development Approaches Working Group WORKPLAN 2011-12 RHSC. HANDtoHAND Campaign Core Messaging RHSC. Monitoring and Evaluation Framework (2007–2015) RHSC. Monitoring Report (2007-2009) RHSC. Monitoring Report (2009-2011) RHSC. Sparking change through the Innovation Fund: Successes and lessons learned for reproductive health supplies RHSC. Strategic Plan (2007–2015) RHSC. Terms of Reference (2009) Systems Strengthening Working Group Workplan: May 2010 thru November 2012 UNFPA (February 2012). AccessRH Metrics as of 31 December 2011 UNFPA (2007). Progress Report UNFPA (2008). Global Programme to enhance Reproductive Health Commodity Security UNFPA (2009). GPRHCS Annual Report UNFPA (2010). GPRHCS Annual Report UNFPA (April 2012). Key Elements: Global Programme to Enhance Reproductive Health Commodity Security (GPRHCS) Phase II UNFPA. Interagency Consultation: Harmonizing approaches to Quality Assurance of Hormonal Contraceptives procured by RHSC partners 68 Evaluation of the Reproductive Health Supplies Coalition USAID (2009). Contraceptive Security index 2009: a tool for priority Setting and planning USAID (2009). Market Development Approaches Working Group USAID (2010). Developing a Family Planning Goal for 2015 that Supports MDG-5b USAID (April, 2008). Commodity Security for Essential Medicines. Challenges and Opportunities USAID. Contraceptive Security Ready Lessons WHO (2010). Progress Report 2010 Reproductive Health Essential Medicines: achievements, lessons learnt and next steps WHO (2011). List of Prequalified Medicinal Products 69 Evaluation of the Reproductive Health Supplies Coalition Summary: This paper details the workplan for the evaluation of the RHSC. This is the initial Deliverable under the consultants’ contracts with the Gates Foundation. The workplan contains the methodology for the evaluation (Item 1), a list of Documents and Materials Reviewed to date (Item 2), a list of organizations from which key informants are being drawn (Item 3), overarching questions which will frame the evaluation (Item 4) and an evaluation timeline (Item 5). Prior to this workplan being developed in detail the team met with the RHSC donors in London on March 15th. The following key points arose from that meeting: • The donors recommended that the evaluators should add to the methodology a brief survey to be sent electronically to all RHSC members. The team has taken this on board (see Item 1c) • Based on donor interactions during the meeting with key questions from the SOW, the evaluators will also derive a concise set of overarching questions. These overarching questions will be the frame for this evaluation (see Item 4). • The donors agreed to meet the evaluators in New York (in person and/or by video conference), provisionally set for 11am EST on May 11th, for presentation and discussion of preliminary findings emerging from the evaluation. • The donors agreed that their common point of primary interest was the future of the RHSC, particularly in terms of its role in the changing environment of commodity security (e.g., the Gold Moment for Family Planning), its potential for encouraging collaboration among partner organizations to advance CS, as well as family planning more broadly, within mid and low income countries. • The evaluation team flagged the lack of up-to-date results against indicators in the M&E framework which may adversely affect their ability to make evidence based conclusions about the impact of the RHSC. • Similarly, meeting participants noted the difficulty of establishing value for money for this type of initiative. Introduction: Don Lauro and Adrienne Chattoe-Brown are pleased to submit this Workplan for the RHSC Evaluation. This is the initial Deliverable under their consultant contracts with the Gates Foundation. The evaluators developed this Work Plan to accord with the RHSC Evaluation Statement of Work (SOW), produced collaboratively by the donors supporting RHSC: Department for International Development (DFID), the Bill and Melinda Gates Foundation (BMGF), United Nations Fund for Population Activities (UNFPA), and the United States Agency for International Development (USAID). The evaluators wish to thank donor representatives Nel Druce (DFID), Monica Kerrigan and Jose Rimon (BMGF), Kechi Ogbuagu (UNFPA), and Alan Bornbush (USAID) for the overall guidance they provided for this Work Plan during a meeting held on March 15th in London. This Work Plan contains the following five items: 1) Methodology for the Evaluation 2) Documents and Materials Reviewed 3) Key Informants (organizations) 4) Overarching Questions to Frame the Evaluation 5) Evaluation Timeline Annex 3 RHSC Evaluation Workplan 70 Evaluation of the Reproductive Health Supplies Coalition Item 1: Proposed Methodology for the RHSC Evaluation a) Document Review: The RHSC Evaluation began with a review of a comprehensive set of RHSC and related contraceptive security (CS) documents. As directed by the SOW, many of these materials were gathered from the RHSC website (www.rhsupplies.org). In addition, the evaluators identified a number of documents from other sources. Documents reviewed include internal RHSC and Working Group documents, meeting notes, commissioned studies, articles, published papers; program reports; unpublished technical papers, reports, data compilations, and presentations; video; and multi-media presentations. Some key informants may also alert the evaluators to additional materials relevant to RHSC, implementing partner organizations, and others working on contraceptive security and related issues. Documents reviewed to this point are listed in Item 2 of the Work Plan. A list of all documents reviewed and found pertinent to this evaluation will be included as an appendix of the Evaluation Report. b) Interview Key Informants: A key informant is primarily defined as someone who has breadth as well as depth of knowledge about RHSC. As per the SOW, the total number to be interviewed will be 45 key informants. Most key informants are within the following: donor organizations supporting or potentially funding RHSC and CS; the RHSC Secretariat in Brussels; RHSC Executive Committee members; heads and members of RHSC Working Groups; and partner organizations working in collaboration with RHSC. Some key informants however may not be directly connected with RHSC but offer broad perspectives on this collaborative approach, alternative models, and/or CS. The Evaluators compiled a preliminary list of key informants and organizations and discussed this with the donor group at the March 15th meeting in London. Based on input provided at this meeting, the donors added to the initial list. A few additional key informants may be added to the list with further inputs from the donor group, the RHSC Secretariat, and others as the evaluation proceeds. (For the current list of organizations from which key informants are being drawn, see Item 3). A final list of key informants by organization will be appended to the Evaluation Report. To collect relevant data from key informants, the evaluators will conduct semi- structured interviews using open-ended questions. Based on the SOW, background documents, and guidance provided by the donor group, for each interview or set of interviewees, key questions will be extracted from a longer generic questionnaire. The evaluators will develop this list of questions in relation to the three objectives and 19 associated key questions specified in the SOW. As discussed and agreed at the meeting with donors, the questionnaire and findings of this evaluation will be framed within eight overarching questions – see Item 4. A generic questionnaire from which questions will be selected for key informant interviews is currently being drafted, and will be appended to the Final Report. In order to elicit information relevant to completing this SOW, the evaluators, as appropriate for particular key informants, may also ask other questions that may not be on this list. In-person interviews of key informants are preferred. For this reason, a number of key informant interviews will be scheduled for the period March 21st to April 4th when both evaluators will be in Europe and April 30th to May 11th when both evaluators will be on the East Coast of the US. Residential proximity of the evaluators to other key informants based in the UK and California may afford opportunity for additional face- to-face interviews. However, as needed, remote phone or email interviews will be used to interview other key informants, including those working outside of Europe or the US, notably within the countries and regions where RHSC partners are active. 71 Evaluation of the Reproductive Health Supplies Coalition Key informants include the Executive Committee Chair and members, donor group representatives, RHSC Secretariat director and staff, and representatives of other organizations who have current or previous involvement with RHSC. The evaluators will make particular effort to interview former and current Working Group heads as well as a selection of participating members. The evaluators will also give some special attention to some donors, particularly ones who benefit from and may potentially contribute to RHSC-like operations in the future. In addition to key informants from within mid and low income countries who are interviewed, the evaluators will also more broadly solicit input from partners in the Global South by virtue of an electronic survey of RHSC members (see next sub-section). Gathering information from selected key informants may require more than one interview. This may be the situation with some from the donor group as well as within the RHSC Secretariat and Executive Committee. Such occasions will enable the evaluators to focus on and dig deeper into various objectives and questions in the SOW, particularly as data collection and analysis proceeds. The list of organizations from which key informants will potentially be drawn, for 45 interviews, is in Item 3. Members of the Secretariat are an important set of key informants, with whom the evaluation team will spend some time, both individually and as a group, during a visit to the Secretariat in late March. In addition to covering the evaluation questions, the evaluators will ask the Secretariat to clarify issues of importance for this evaluation. The team will also request an update, to the extent possible, of the Monitoring and Evaluation indicators. In addition, the evaluators may engage Secretariat staff in defining, diagramming, and updating RHSC’s theory of change. c) Survey of RHSC Members: Using Survey Monkey or another internet-based survey tool, the evaluators will design a short questionnaire for RHSC to send to all member organizations. The purpose of this is to ensure that all members are offered the possibility of providing direct input for this evaluation. It will be the responsibility of the evaluators to interpret tabulations of data that result and incorporate these into findings of the evaluation. This survey was not specified in the SOW but is being included at the suggestion of the donor group as a useful adjunct to the evaluation. d) Participation/Observation: On occasion, as opportunity and time allow, the evaluators will actively participate or observe RHSC activities. For example when possible and depending on occasions that arise, the evaluators may attend relevant meetings or passively participate in Working Group conference calls (e.g., the meeting of the MDA Working Group to be held in the San Francisco area in May and monthly conference calls that some Working Groups hold). Such opportunities will enable the evaluators to collect a different level of data than that yielded by interviews or literature review. However, opportunities to do so may be limited by time, circumstance, and logistics. E) Other data sources: The Scope of Work calls on the evaluators to “assess how well the RHSC has met its objectives and made an impact on RH commodity security”. The ideal way to do this would be to rely primarily on progress against the indicators in the Monitoring and Evaluation Plan for the Strategic Plan (2007–2015). However the most recent monitoring report is dated 2009 and the retrospective aspects of evaluation are meant to focus primarily on the period since that date. The team has explored with the Secretariat the possibility of receiving an up-to-date 72 Evaluation of the Reproductive Health Supplies Coalition report against these indicators but understand that this is a considerable effort involving the contribution of many partners. Whilst the team will explore this further with the Secretariat when they meet face to face, it may not be possible to get all that data. If this is the case the team will consider whether alternative measures can be used. The consultants flagged this with the donor group at the March 15th meeting. F) Assessing value for money: The Scope of Work calls on the evaluators to establish “To what extent, and to the extent that it can be gauged, does the RHSC achieve value for money in terms of activities undertaken and results achieved?” This was raised with the donor group and it was agreed that this is particularly difficult for this type of initiative. The team will continue to consider how this can be approached. g) Data Analysis Plan: Questions for the interviews of key informants will be coded in relation to the three Evaluation objectives, the associated key questions in the SOW, and the overarching questions that the evaluators developed to frame this evaluation. Some responses to interview questions will be concurrently recorded (voice recordings) and saved as electronic files. The recording are then transcribed and subsequently edited for relevancy. Stored electronically, these files will be further coded for particular themes and issues relevant to the evaluation. These files, like the interviews themselves, are considered confidential. Attribution to individuals will only occur with prior permission. To analyze key informant responses, the evaluators will both comprehensively review all responses and conduct separate content searches for key terms and concepts. Based on this, the evaluators will identify common and/or significant issues in relation to questions posed in the SOW. Preliminary summations in terms of SOW and overarching questions will be checked against information from project documents and other sources, including the member survey and opportunities to observe or participate in RHSC virtual or actual activities. Triangulation will occur in terms of what documents indicate RHSC intended to and/or did accomplish, what key informants reported about RHSC, and what the evaluators learned from the combination of document review, interviews, the web-based member survey, and occasions to observe RHSC in action. As encouraged by the donor group, analysis will give special attention to future prospects and potential for collaboration among partner organizations to advance CS, as well as family planning more broadly, within mid and low income countries. 73 Evaluation of the Reproductive Health Supplies Coalition ITEM 2: Documents and Materials Reviewed Document Title Web location Summary & comments Coalition core documents RHSC Terms of Reference: Organizational Principles and Structural/Governance (June 2009 http://www.rhsupplies.org/fileadmin/us er_upload/Strategy/TOR_2__3June2 009.pdf Essential background document (2009) that covers the operational rules or terms as well as structure of RHSC. Update of 2005 version. Revised to include recent developments. Reproductive Health Supplies Coalition Strategic Plan (2007– 2015) http://www.rhsupplies.org/fileadmin/us er_upload/Strategy/Strategic_Plan_2 007.pdf This five-year old document benchmarks what RHSC had laid out five years ago as its expected aims and anticipated accomplishments. The RHSC Strategic Plan is based upon working group inputs to extend what was in the original TOR formulation. There is an earlier version. Reproductive Health Supplies Coalition Monitoring and Evaluation Framework (2007–2015) http://www.rhsupplies.org/fileadmin/us er_upload/Strategy/ME_Framework.p df Building on the Strategic Plan, RHSC has produced this M&E Framework, which includes 30 indicators aligned with the three goals: increase resources; strengthen systems; and add value. These are usefully presented in accordance with a schematic overview of the Strategic Plan (p.5) and detailed in a document appendix. Reproductive Health Supplies Coalition Monitoring Report (2007–2009) http://www.rhsupplies.org/fileadmin/us er_upload/MR2009_2.pdf This document provides an initial tracking of M&E indictors over a two-year period. Need an update to the present time and an understanding of how such data (i.e., this report) is routinely compiled by RHSC. Access for All: The Secretariat of the Reproductive Health Supplies Coalition Annual Report -- 2010 http://www.rhsupplies.org/fileadmin/us er_upload/Annual_Report/ARFinal.pd f This pleasingly formatted report provides a useful update on RHSC and partner activities to 2010. Coalition working groups RHSC Working Groups http://www.rhsupplies.org/working- groups.html Portal to much additional information on the RHSC Working Groups (workplans, meeting archives, members, Innovation Fund, FAQs, etc.) as well as other partnering structures: the Caucus on New and Underused RH Technologies, and regional groups for LAC and Francophone 74 Evaluation of the Reproductive Health Supplies Coalition Document Title Web location Summary & comments MDA http://www.rhsupplies.org/working- groups/market-development- approaches.html Webpage for working group Market Development Approaches Working Group WORKPLAN 2011-12 http://www.rhsupplies.org/fileadmin/us er_upload/MDA_Documents/MDAWG _Workplan_2011-12_FINAL.pdf Contains revised and updated workstreams (activities) for the coming year – a product of a two-day meeting in November, 2010. Useful as an outline of this working groups efforts, but does not provide great deal of information on how this work gets funded and carried out. Market Development Approaches Scoping Report http://www.rhsupplies.org/fileadmin/us er_upload/MDA_Documents/MDA_Sc oping_Report_2006_10.pdf This 2006 study was commissioned by the Gates Foundation and conducted by HLSP through literature review and 28 key informant interviews, mostly with the Market Development Approaches Working Group. Includes several useful diagrams that clarify MDA. Market Segmentation Primer http://www.rhsupplies.org/fileadmin/us er_upload/MDA_Documents/MDAWG _Market_Segmentation_Primer_FINA L_doc.pdf Conceived by the MDAWG, this primer provides basics on segmentation studies, including what, how, where (case studies), and why. MDA indicators http://www.rhsupplies.org/fileadmin/us er_upload/MDA_Documents/WS2_To tal_Market_Indicators_SK1.doc From 2008. MDA current use of innovation fund TMI Honduras http://www.rhsupplies.org/fileadmin/us er_upload/Kampala/presentations/Ril ey_Pam_-_TMI_Honduras_2010-05- 27.ppt Interesting example. Innovation fund paid for Abt and JSI to do market analysis and bring stakeholders together. TMI Madagascar http://www.rhsupplies.org/working- groups/market-development- approaches/innovation-fund- activities/total-market-initiative- madagascar.html Supported under the Innovation Fund (Round Two) this study of total market approaches was used to advocate with the government for reducing taxes on contraceptives and being open to private sector involvement. MSI, Futures and UNFPA implemented it. What exactly did IF pay for? 75 Evaluation of the Reproductive Health Supplies Coalition Document Title Web location Summary & comments Accessing quality assured RH supplies http://www.rhsupplies.org/working- groups/market-development- approaches/innovation-fund- activities/accessing-quality-assured- rh-supplies.html Funding to Concept Foundation to help prequalification of manufacturers of hormonal contraceptives. Medicines for Reproductive Health: Ensuring Access to Quality Assured Products http://www.rhsupplies.org/fileadmin/us er_upload/Access/Concept_Medicine s_for_Reproductive_Health.pdf Funded by the innovation Fund and authored by Peter Hall and Lester Chinery of the Concept Foundation, this paper analyzes generic RH medicines and the WHO Prequalification Program. It usefully reviews the changing circumstances within which contraceptives are procured, including the overall industry shift to generics and developing countries’ increasing reliance on hormonal generics of uncertain quality. RMA Workplan 11/12 http://www.rhsupplies.org/fileadmin/us er_upload/RMA_WG_Documents/RM AWG_Workplan_2011-12.docx Very brief workplan for the working group. Contraceptive Projections and the Donor Gap: Meeting the Challenge http://www.rhsupplies.org/fileadmin/us er_upload/RMA_WG_meetings/RHS C-FundingGap-Final.pdf This RHSC commissioned 2009 study by John Ross, John Stover, and Eva Weissman updates earlier studies that show a gap between projected funding and prospective needs for contraceptives. Based on analysis of data from 88 countries, donor funding for commodities (which has largely been condoms, pills, and injectables) would have to increase by $220 million to a total of $450 million to meet unmet need. The Road from Istanbul to Addis and Beyond: Setting and Agenda for Reproductive Health Supplies http://www.rhsupplies.org/fileadmin/us er_upload/Access/The_Road_from_Is tanbul_to_Addis_Ababa.pdf This 2011 PAI produced report summarizes progress and perspectives gathered by interviews within six countries: Bangladesh, Ethiopia, Ghana, Mexico, Nigeria, Tanzania. Supported by innovation round 5. RMA current use of innovation fund 76 Evaluation of the Reproductive Health Supplies Coalition Document Title Web location Summary & comments RH advocacy and 2010 G8 http://www.rhsupplies.org/working- groups/resource-mobilization-and- awareness/innovation-fund- activities/rh-supply-advocacy-and- 2010-g8.html “Using Canada’s 2010 presidency of the G8 to heighten global awareness of Canada’s contribution to commodity security, and to increase Canadian government support for the issue and for the work of the Coalition.” Engaging Francophone Africa http://www.rhsupplies.org/working- groups/resource-mobilization-and- awareness/innovation-fund- activities/engaging-francophone- africa.html “To raise awareness of supply security in francophone Africa and to strengthen the French-language networks, communications media, and other tools needed to meet countries' supply needs”. Developing a documentary on RH Supplies http://www.rhsupplies.org/working- groups/resource-mobilization-and- awareness/innovation-fund- activities/developing-a-documentary- on-rh-supplies.html “Producing and promoting an advocacy film, which is designed to put a human face on the issue of reproductive health supplies. showcase the work of Coalition members and partners, and provide advocates with a powerful means to communicate key advocacy messages.” Systems strengthening Workplan 11/12 http://www.rhsupplies.org/fileadmin/us er_upload/SSWG/SSWG_W- Plan_2010-12.pdf Several important initiatives in this group Countries at risk Info sheet on data coordination for Countries at Risk. Gives examples of responses to information. AccessRH http://www.rhsupplies.org/fileadmin/us er_upload/SSWG/AccessRH/Access RH_background.pdf Short explanation of AccessRH Current Innovation grant http://www.rhsupplies.org/working- groups/systems- strengthening/innovation-fund- activities/incorporating-manufacturer- input-into-accessrh.html JSI - soliciting manufacturer input into the operational design of AccessRH Pledge guarantee http://www.rhsupplies.org/working- groups/systems-strengthening/the- pledge-guarantee-for-health.html This RHSC website description is also linked to FAQs and press announcements. As of 2011 there were two proof-of- concept applications of the Pledge Fund: one in Zambia and one in Ethiopia. 77 Evaluation of the Reproductive Health Supplies Coalition Document Title Web location Summary & comments Fact Sheet http://www.rhsupplies.org/fileadmin/us er_upload/SSWG/PGH_- _Informational_Brochure.pdf Fact sheet on pledge guarantee. Explains how it works The Pledge Guarantee for Health: A New Platform to Improve Efficiency and Impact of Donor Financing for Health Commodities http://en.fininnov.org/img/pdf/ateliers/ The%20Pledge%20Guarantee%20for %20Health.pdf This UNF PowerPoint displays the main operating features and advantages of the Pledge Guarantee for Health Professional development of supply chain managers http://www.rhsupplies.org/fileadmin/us er_upload/SSWG/RHSC_Professiona lizing_SCM_White_Paper_1.pdf A ‘white paper’ which proposes goals, an approach, activities, and a call to action that aims to engage key stakeholders in a harmonized effort to professionalize public health supply chain management Current innovation grant http://www.rhsupplies.org/working- groups/systems- strengthening/innovation-fund- activities/professional-development- opportunities-i-and-ii.html JSI mapping programmes, identifying needs and making recommendations Prequalification http://www.rhsupplies.org/working- groups/systems- strengthening/prequalification.html Prequalification overview Lists of pre-qual commodities http://www.rhsupplies.org/working- groups/systems- strengthening/prequalification.html WHO, condoms and IUDs. Some of the European manufacturers are coalition members. Pre qual meeting report Harmonization meeting report RH Interchange http://rhi.rhsupplies.org/rhi/index.do?l ocale=en_US A brief explanation of this website portal to data on commodity orders and shipments, with link to how data is compiled. (Developed within JSI, RH Interchange now operates under UN auspices in Copenhagen.) RH supplies in emergencies http://www.rhsupplies.org/working- groups/systems- strengthening/reproductive-health- supplies-in-emergency-settings.html Webpage suggests there should be a report, and that the activity is finished. But no report referenced. 78 Evaluation of the Reproductive Health Supplies Coalition Document Title Web location Summary & comments Users guide to supply chain software http://www.rhsupplies.org/working- groups/systems-strengthening/a- users-guide-to-supply-chain- software.html Webpage suggests there should be a report, and that the activity is finished. But no report referenced. Other Innovation fund grants (current) Supply chain mgt of comm. Based distribution programmes http://www.rhsupplies.org/working- groups/systems- strengthening/innovation-fund- activities/supply-chain-management- for-cbd-programs.html JSI implementing: Describing current supply chain management practices and utilization of CBD in a number of countries and programs. Analyzing trends and current practices for supply chain management of CBD programs. To be used to develop best practices. Improved RH supplies procurement in LAC http://www.rhsupplies.org/working- groups/systems- strengthening/innovation-fund- activities/improved-rh-supplies- procurement-in-lac.html Object – to build the capacity of procurement staff in the Latin America/Caribbean (LAC) region LAC Forum overview Translation of the webpage. Caucus Caucus on new/underused RH technologies http://www.rhsupplies.org/working- groups/caucus-on-newunderused-rh- technologies.html Participation in this cross-cutting forum requires membership in one of the Working Groups and serves as an intermediary with implementation occurring through Working Group activities. RHSC Country Profiles http://www.rhsupplies.org/resources- tools/country-profiles.html Country profiles/reports under RHSC focus countries (included in M&E framework). This includes countries in Africa, Asia and Latin America and provides updated quantitative and qualitative information about the each country’s progress. The Innovation Fund http://www.rhsupplies.org/resources- tools/innovation-fund.html Funded by the Gates Foundation, this Fund allows Working Group members to access support for particular projects and studies, some 16 of which have been conducted to date. Hand to Hand Campaign 79 Evaluation of the Reproductive Health Supplies Coalition Document Title Web location Summary & comments Hand to Hand Campaign Factsheet http://www.rhsupplies.org/fileadmin/us er_upload/Handtohand/HANDtoHAN D_Campaign_Core_Messaging.pdf Succinct, core messaging statements on the HandtoHand Campaign Understanding 100 million http://www.rhsupplies.org/understandi ng-100-million.html Provides background in terms of unmet need for the Hand-to- Hand Campaign to reach 100 million new FP users by 2015 Developing FP goal http://www.rhsupplies.org/fileadmin/us er_upload/Handtohand/BRIEF_publis hed.pdf USAID note on developing a FP goal for 2015 that supports MDG5b. 2010. Other Coalition outputs Video: Meeting the Challenge of a New Decade-Reproductive Health Commodities http://www.rhsupplies.org/en/addis- ababa-2011/proceedings/popup- film.html This well-done video presents a well-illustrated overview of the past decade of meeting reproductive health commodity security from the launch of the Istanbul conference in 2001 to the HandtoHand Campaign that calls for joint international advocacy to raise awareness, political support and funding commitment to ensure commodity security and access to FP. RHSC briefly mentioned as the glue that holds this together. MyAccessRH.org http://www.myaccessrh.org/ Through UNFPA, AccessRH offers convenient access to a wide range of quality, affordable reproductive health products, as well as up-to-date information on over US $1.6 billion of global contraceptive orders.The AccessRH concept was developed by the RHSC, is managed by UNFPA, and funded by the Gates Foundation, the EU, the German Federal Ministry for Economic Cooperation and Development (BMZ), and thru in-kind support from USAID. Other documents 80 Evaluation of the Reproductive Health Supplies Coalition Document Title Web location Summary & comments The World We Want – Beyond 2015: A Toolkit for National Deliberations http://whiteband.org/sites/default/files/ B2015%20Toolkit%20Final.pdf A joint publication (GCAP, Beyond 2015, and UN Millenium Campaign) outlines process to promote and ensure CSO and national-level participation in next iteration of MDGs. Covers contemporary strategies and actions toward a process for building a post-2015 consensus. In light of original MDGs absence and then initial neglect of FP/RH, important for considering how FP/RH, including CS, will be positioned in the iteration of international consensus towards addressing and alleviating poverty and inequity. G8 Muskoka Initiative: Maternal, Newborn, and Under-Five Child Health No link. Is this the right one? http://www.g8.utoronto.ca/summit/201 0muskoka/communique.html#annex1 This two-page 2010 document reaffirms commitment of the G8 to MDGs 4 and 5, specifically referencing unmet need for FP, and includes “sexual and reproductive health care and services, including voluntary family planning”. Commodity Security for Essential Medicines: Challenges and Opportunities http://pdf.usaid.gov/pdf_docs/PNADM 797.pdf This 2008 Report, by USAID’s Deliver Project, provides an overview of the challenges and opportunities to attain commodity security and presents different frameworks. Building on WHO’s EML model, the report makes the case for integrating CS into the larger context within which supplies are part of health system improvements. Progress Report 2010 Reproductive Health Essential Medicines: Achievements, Lessons Learnt and Next Steps http://whqlibdoc.who.int/hq/2010/WH O_RHR_10.23_eng.pdf This report highlights the contributions of many stakeholders’, including RHSC, role in contributing to the WHO Reproductive Health Essential Medicines Program to “lay the groundwork for more effective public-sector procurement of good-quality reproductive health medicines and commodities” and to contribute to a “harmonized list of reproductive health medicines into the WHO Model List of Essential Medicines”. This report concludes with challenges and next steps. USAID Summary of Support for Contraceptive Security http://www.usaid.gov/our_work/global _health/pop/techareas/contrasecurity/ index.html link not working A portal to 10 lessons learned within CS, from raising awareness to integrating supply chains. 81 Evaluation of the Reproductive Health Supplies Coalition Document Title Web location Summary & comments Contraceptive Security Ready Lessons: Overview http://pdf.usaid.gov/pdf_docs/PNACW 660.pdf Provides and introduction and overview to the full Contraceptive Secrity Ready Lessons Report available on the USAID website. UNFPA Global Programme on RHCS 2007 – 2010 annual reports The link between the GPRHCS and the RHSC is (a) the secretariat funding comes out of this programme and (b) the GPRHCS is sometimes regarded as the commodity funding arm of the SC. Programme document Contains details of how the Programme will work Useful docs Reproductive Health Commodity Security: Leading from behind to forge a global movement Interesting history of the coalition and what came before. Item 3: Key Informant Organizations Key informants (one or more) will be drawn from a shortlist of the following organizations. Other informants may be added later in the evaluation process (whilst still remaining within the 45). Governmental organisations • China Contraceptive Supply Administration (CCSA) • East African Community (EAC) • Ethiopia - Ministry of Health • France - Ministry of Foreign and European Affairs • Germany - Federal Ministry for Econmic Cooperation and Development (BMZ)/KfW Development Bank • Global Fund to fight AIDs, Tuberculosis and Malaria • Netherlands - Ministry of Foreign Affairs • Partners in Population and Development • Uganda - Ministry of Health • United Kingdom - Department for International Development • United Nations Population Fund • United States Agency for International Development • West African Health Organisation (WAHO) • World Bank • World Health Organization/Essential Drugs Management Commerce and industry • Bayer Pharma AG • Famy Care Limited • Female Health Company • Helm Pharmaceuticals GmbH • MSD (Merck Sharp & Dohme Corp.) 83 Evaluation of the Reproductive Health Supplies Coalition • Shanghai Dahua Medical Apparatus Co., Ltd. • Foundation and Institutions of Higher Learning • Bill & Melinda Gates Foundation • Bill and Melinda Gates Institute for Population and Reproductive Health, Bloomberg School of Public Health, John Hopkins University • David and Lucile Packard Foundation • United Nations Foundation • William and Flora Hewlett Foundation Technical Support Agencies and Partnerships • Advocacy Nigeria • Asociación Benéfica PRISMA • Concept Foundation • Countdown 2015 Europe • DKT International • DSW • Greenstar Social Marketing Pakistan • Gynuity Health Projects • Ipas • John Snow, Inc (inc DELIVER) • Marie Stopes International • PATH • Population Council • Population Services International • Venture Strategies for Health and Development (VSHD) Civil society • Equilibres & Populations 84 Evaluation of the Reproductive Health Supplies Coalition • Family Guidance Association of Ethiopia • Interact Worldwide • International Planned Parenthood Federation • Population Action International • Reproductive Health Uganda • Women Deliver Others • Margaret Verveik • RHSC secretariat Item 4: Overarching Questions to Frame the Evaluation Following an analysis of key questions in the SOW, the donor group agreed that the evaluators would include in the work plan a smaller subset of overarching questions to frame the evaluation. These framing questions are provided below. Each question is linked with one of the three SOW objectives: A. Accomplishment; B. Structure; and C. Future Direction. In addition, these overarching questions are linked to the numbered key questions included in the SOW. This organization will enable the interviews to develop a generic questionnaire for key informant interviews that derives from and is consistent with the SOW. In addition these questions provide the basis for organizing the presentation of findings from this evaluation. OVERARCHING QUESTIONS What contributions has RHSC made and what impact has it had on improving contraceptive security at global, regional, and country levels? SOW question: A-1 Over and above partner organizations’ own accomplishments, what is RHSC’s added-value and comparative advantage and value-for-money? SOW questions: A-3 and B-1 To what extent is RHSC’s role, and especially that of the Secretariat, adequately reflected in its theory of change and M&E framework? SOW questions: A-2, B-6, and C-1 In what ways and to what effect has the structure and operation of RHSC evolved, particularly since 2010? 85 Evaluation of the Reproductive Health Supplies Coalition SOW questions: B-4, B-7, and B-11 What works well and what needs to be improved or changed about RHSC’s internal structure and external operations? SOW questions: B-3, B-8, C-1 (bullet), and C-3 To what extent has RHSC been responsive to partner-led initiatives or and proactive towards new opportunities? SOW questions: B-2, and B-5 Does RHSC seem poised and/or have potential for improving and increasing contraceptive security in the future? SOW questions: C-5, C-6, and C-7 Beyond serving as an umbrella for collaboration, what alternative models or approaches may be needed in the coming years to advance contraceptive security globally, regionally, and at country level? SOW questions: C-2 and C-4 Item 5: Evaluation Timeline Month – Dates Tasks March 1-9 Preparation for assignment: Orientation to assignment and review of background materials (SOW, RHSC website, and related materials (e.g., Working Group Work Plans and Reports; CS documents and articles, etc.) March 12-16 In London, evaluators coordinate documentation review and begin preliminary work on Evaluation Work Plan. On March 15th, evaluators meet and work with donor group representatives (Monica Kerrigan and Jose Rimon of the Gates Foundation; Nel Druce of DFID; Alan Bornbush of USAID; and Kechi Ogbuagu of UNFPA). March 19 – April 6 Evaluators draft the Work Plan and send to donor group by March 20th; based on SOW questions, evaluators develop overarching questions to frame evaluation findings and a generic questionnaire for key informant interviews; evaluators schedule interviews, begin conducting interviews in London, and travel to Brussels (March 26th) and other locales in Europe to interview key informants April 9 - 18 Evaluators schedule and conduct phone interviews with key informants not available for in-person interviews, including in selected countries and regions where RHSC members are active. Evaluators also schedule interviews for upcoming visit to DC and NYC. Evaluators continue phone interviews and in-person interviews on the West Coast or in UK. April 26 – May 11 Both evaluators travel to DC and NYC where they interview key informants from donor and member organizations. Concurrently interview data are organized into electronic files to facilitate subsequent analysis. After synthesizing data, evaluators develop preliminary findings and organize these in PowerPoint. On May 11th (subject to 86 Evaluation of the Reproductive Health Supplies Coalition confirmation by the donor group), evaluators meet with the donor group in NYC to present and discuss preliminary findings. May 14 – May 30 Evaluators continue analyzing data, draft Evaluation Report, and deliver draft Report to donor group by May 30. May 31 – June 13 Donor Group representatives review draft Report and provide feedback to evaluators by June 13th By date of Executive Committee Meeting Evaluators incorporate donor input to revise draft Report and send to RHSC Secretariat for consultation. June 17-22 Evaluators travel to London and present Evaluation findings to Executive Committee of RHSC at their Spring 2012 meeting June 25-29 Finalization of Evaluation Report 87 Evaluation of the Reproductive Health Supplies Coalition A. Achievement 1. To what extent has the RHSC fostered progress for RH commodity security at all levels (global, regional, country), and, to the extent that it can be gauged, what has been the impact of the RHSC on increasing access to quality, affordable RH supplies for clients in low and middle income countries? • What progress has the RHSC made specifically against its Strategic Plan, as measured by the Monitoring and Evaluation Framework? • How successful has the RHSC been in increasing the prominence of RH commodity security in global health and, more broadly, international development efforts? • How successful have RHSC-inspired initiatives been in terms of country uptake and results? This question closely mirrors and the answer has largely been addressed in Section 3 of this report. While there have been considerable global-level achievements, regional level is at this point more promising than actual, and country-level impact, as available data show and key informants widely report, has been the least of what can be claimed as attributable to the Coalition. There are some exceptions, in terms of stock-outs averted or adumbrated; advocacy in a few countries; and price reduction for one major commodity in Ethiopia. In general, however, while the RHSC has become well positioned and widely respected as a voice for FP/RH commodity security at global level, country level remains an area for future attention. Given membership growth in this direction, as well as participation of two regional forums as Coalition members, there is clear potential for the RHSC to do more to have greater impact at this level in the future. At global level, long-term efforts, initially by the Supplies Initiative and since 2006 by the RHSC, have clearly placed commodities more centrally within the consciousness of the broader field of FP/RH. Members interviewed as key informants, as well as many who responded to the electronic survey of members, acclaimed that the RHSC has raised the profile of commodity security within the international arena. Of equal importance are contributions made to advance knowledge and understanding of product logistics and availability within the critical path of ensuring that choices are available to all who wish to use them. This is not to say that RHSC’s impact has been documented and well measured. Rather, the Coalition is valued by member organizations for bringing attention to and building knowledge about a critical and erstwhile neglected part of FP/RH – the products needed to make programs work. The evaluators requested and received from the RHSC an update on the indicators contained within the Monitoring and Evaluation Framework. (For details, please see the tables in Annex 13 also included in this Report.) In general, though the indicators do not completely map against what would be needed to determine impact, the data show and the evaluators concur that progress at the global level has been considerable. Regional progress is not tracked by the indicators currently used. However, inclusion of two regional forums as members of the Coalition is promising and has potential for future advances in FP/RH commodity security in the LAC and Francophone Africa regions. However, the evaluators conclude and available data and Annex 4 Answers to Key Questions from RHSC Scope of Work 88 Evaluation of the Reproductive Health Supplies Coalition other evidence affirm with country-level impact attributable even broadly to the RHSC has been considerably less. The structure and operations of the RHSC are such that it is difficult to tease out the exact contribution of the Coalition as a whole, rather than the influence of individual members. However there are examples where there has been demonstrable added value of the Coalition to the global health agenda. For example efforts by Coalition members working together helped to get family planning commodities included in the UN Commission on Life Saving commodities for Women and Children. The RHSC then contributed directly to the work of the Commission by advising on the selection of commodities to be included, and carrying out data collection and analysis. The Coalition has also helped to bring provision of commodities into the maternal health debate by the development of the Hand to Hand Campaign. Part of the rationale for its development was to rally the support of the family planning community around the maternal health agenda in particular around “Every Women/Every Child”. The Campaign can be seen as the forerunner of the July 2012 Family Planning Summit which represents a widening realization of the importance of contraceptives to achieving MDG 5 and indirectly MDG 4. Generally the RHSC could be said to be helping to move family planning into a more center stage position where previously it had been sidelined subsumed into wider agendas such as maternal health and HIV/AIDS and as a consequence had lost focus. It has also helped reduce the gap that has existed between the family planning and maternal health communities. The RHSC has also helped to demonstrate the potential contribution of manufacturers to the wider development agenda, positioning them as partners in family planning rather than adversaries, as has been the case in some global health initiatives. This has relevance to the work of UNITAID, Clinton Health Access initiative and bilateral donors in the area of access to medicines. The Secretariat has exercised leadership by understanding and opening discussions with various pharmaceutical manufacturers, at first in the face of opposition to them being members of the Coalition. Useful understanding is emerging that the end-game is not just about lowering prices but also sustainability and availability of products. 2. Does the RHSC have and has it worked according to a demonstrably effective “results chain” or theory of change, where the Coalition’s operational strategies (collaboration, advocacy, innovation, technical solutions) have guided inputs that in turn have translated into outputs and outcomes for RH commodity security? The short answer to this question is straightforward: No, the RHSC does not have a results chain or theory of change. The strategy for the program, the 2007-2015 RHSC Strategy, as well as the 2009 Terms of Reference, define the operations and describe the structure within which the RHSC operates. The RHSC has also usefully summarized its strategy in an interactive 3x3 matrix on its website. However, the RHSC does not have a theory of change. More importantly, it has not updated its strategy since the original formulation. “A theory of change (TOC) is a tool for developing solutions to complex social problems. A basic TOC explains how a group of early and intermediate accomplishments sets the stage for producing long- range results. A more complete TOC articulates the assumptions about the process through which change will occur and specifies the ways in which all of the required early and intermediate outcomes related to achieving the desired long-term change will be brought about and documented as they occur.” -- Adapted by Erin Harris, Research Analyst, Harvard Family Research Project, from Anderson, A. (2005). The community builder's approach to theory of change: A practical guide to theory and development. New York: The Aspen Institute Roundtable on Community Change (see: http://www.hfrp.org/evaluation/the-evaluation- exchange/issue-archive/evaluation-methodology/an- introduction-to-theory-of-change) 89 Evaluation of the Reproductive Health Supplies Coalition INGO member staff concerns: First Respondent: I was frustrated by the first strategic plan…I totally thought it was misguided and wrong. Second Respondent: That goes back to the question about the indicators. The indicators represent indicators for a doer rather than. Third Respondent: Exactly!! Oh that’s what you guys were talking about when I came in, you were talking about indicators. Yeah I think that was completely misguided. Second Respondent: There is still a disconnect between what you’re measuring and what you’re doing. Third Respondent: Yeah, what you really are doing! Secretariat Perspectives “Now that we have a better understanding of what the Coalition can do and is doing it makes it possible to develop a strategic plan that reflects… what we see now as some of the outcomes in terms of increased resources and strengthening the systems…with a better sense of where we are adding value.” “A lot of … our M&E indicators are irrelevant or are not as relevant to a coalition as they are to individual members.” The 2007 Program Strategy and the subsequent RHSC Monitoring and Evaluation Framework statically depicts what the Coalition was, not what it has become and currently aspires to achieve. There is clear and recognized need for the RHSC to revisit and revise its strategy and M&E framework. (Quotes) In addition, however, the process of updating its foundational structure would also benefit from parallel development of a theory of change. This would provide the added perspective of logical links between and among anticipated outcomes in the short- term will connect with each other and flow towards long term achievements. As the director and others well recognize, when developed the strategy was largely aspirational, a depiction of what it was hoped that the RHSC could contribute to member accomplishments. Since then a great deal has occurred. A revised strategy to reflect what the RHSC has become would usefully contribute to directions for the future. Developing a theory of change as part of strategic revision would prepare and position the RHSC for further achievement. The process to be used, particularly as a coalition that has experienced rapid expansion of its membership base, would be important to ensuring that cohesion and common understanding among members are maintained. A theory of change complements a program strategy and is often its logical but unarticulated underpinning. A strategy usually defines and delimits activities that a program or project will undertake. A theory of change in contrast links inputs available to causal connections among outcomes to the long term change that is desired. Both are useful. However, particularly for the RHSC as an ongoing venture invested in supporting and advancing a community of practice, a theory of change appears a particularly useful long range tool for foreseeing, managing, and measuring progress. As with strategy development, there are many different and various approaches to formulating a theory of change. The SOW question above about theory of change conveys implicit donor concerns about monitoring and measuring progress towards relevant outcomes within RHSC. Many of those interviewed during this evaluation, including from within the Secretariat, shared these concerns. Recent recruitment of an M&E specialist for the Secretariat should help redress this situation. Focusing on the process as well as the content of a Coalition theory of change will also greatly alleviate shortcomings that the RHSC has had in articulating the change it aspires to make and measuring progress towards those changes. Fostering collaboration underlies how progress is to be made on Goal Three of the strategy. Also referenced in the 2007 RHSC strategy, but not specified as operational strategies, are advocacy, innovation, and technical solutions. However the strategy presentation is linear and interconnections among these and connections with expected outcomes are not made. An alternative approach to consider is the logic model. This is a program planning and management tool used by many funding 90 Evaluation of the Reproductive Health Supplies Coalition organizations. It is more focused and rigorous than a theory of change. As the Kellogg Foundation describes logic model, it provides a “picture of how your organization does its work – the theory and assumptions underlying the program. A program logic model links outcomes (both short- and long-term) with program activities/processes and the theoretical assumptions/principles of the program.” http://www.wisconsin.edu/edi/grants/Kellogg_Logic_Model.pdf) The graphic appeal of theory of change would seem however to make it better suited to a process as well as a product that would be widely shared among members. Two examples of theories of change are shown below: These approaches, theory of change as well as logic models, provide an organization with systematic and visual approaches to develop common understanding of how available resources may be marshalled to achieve results or outcomes that the community is committed to. As such, especially for an organization that is partner-centric, the process of development and revision is as important and useful as the product. Having shared understanding of the relationships among the resources available, activities planned, and the changes or results 91 Evaluation of the Reproductive Health Supplies Coalition expected builds and gives purpose to a community of practice. Within the RHSC, the Program Strategy has not been updated since its original formulation and a theory of change has not yet been attempted. It is time to do both. As important and useful as a theory of change would be as part of the process for revising the RHSC Program Strategy, so may other approaches usefully contribute to preparing the Coalition for its future. Among these, social networking analysis, and in particular mapping of network interactions, may well have useful application for monitoring and understanding Coalition members and their interconnections. This approach, which has notably been applied for mapping leadership and community connections, provides other powerful and promising tools for understanding and monitoring relationships among participating members. 3. What is the RHSC’s comparative advantage? What added value does the RHSC bring to the individual efforts of its members, and more generally to international, regional, and country-level efforts for RH commodity security – over and above what would be accomplished without the RHSC? The evaluators gave considerable attention to assessing comparative advantage, value-added, and over-and-above accomplishments of the RHSC. The neutral space the Coalition occupies as well as its growing stature for convening around FP/RH commodity security are clear headlines that emerge from key informant responses to questions on these subjects. Coalition neutrality combined with leadership exercised at the topmost level has worked most effectively to engage manufacturers, something widely remarked in responses as a concrete example of the added value that the RHSC brings. Similarly, recent engagements with the Chinese and French governments derive from both the neutrality and stature of the RHSC. At regional level, the breadth of Coalition membership combined with its neutrality and convening power made becoming members of the RHSC attractive for two well-placed regional forums, LAC and SECONAF. While the Secretariat has language capability on its staff to help in coordinating with and connecting these regions to other parts of the Coalition, it likely lacks depth of experience that will be needed to assist these forums towards strategic approaches to improving FP/RH commodity security in their respective regions. Thus the value that the Coalition may be able to add at that level is at present limited. Country-level efforts have only received high priority within one of the Working Groups, the RMAWG, and that is more by virtue of funding for complementary projects, Project RMA in the past and Advancing Family Planning at present, than value added by the Secretariat itself or the Coalition more broadly. Furthermore, neither RMAWG-housed efforts nor broader Coalition work at country level can be said to have had significant, widespread impact. Ethiopia, and the contributions that the Coalition made to reducing Implanon prices and connecting procurement to the Pledge Guarantee for Health, is a notable exception. In that country and for that particular effort, RHSC has added value to benefit commodity security and ultimately end users of this method. In other countries, such as Uganda and Ghana, the value added may be more that of the projects promoting capacity for advocacy than that of the RHSC. A number of the charter members of RHSC, from 2004 onwards, had been actively involved with commodity security for many years. Many of these were donors interested in promoting better coordination and collaboration of efforts within contraceptive commodity procurement and purchasing. In 2005, at the Spring Meeting of the RHSC, these donors were joined by representatives of six developing country governments. Unfortunately, this promising start to country-level participation has not been something easily built upon and expanded. 92 Evaluation of the Reproductive Health Supplies Coalition B. Structures/Processes 1. To what extent, and to the extent that it can be gauged, does the RHSC achieve value for money in terms of activities undertaken and results achieved? Have RHSC-managed resources/assets been optimally managed? The RHSC has evolved since 2004. Initially guided by donors, an Executive Committee emerged comprised of donors and other selected members. Since 2006, a small Secretariat has been in place to support the membership and the three Working Groups, also emerging from earlier work, on systems strengthening, resource mobilization, and market approaches. By 2009, RHSC had extended membership to all organizations interested and aligned with the RHSC’s vision that “all people in LMI countries are able to access and use affordable, high- quality supplies, including a broad choice of contraceptives, to ensure better reproductive health.” In 2009 the funding base of the RHSC expanded beyond the Gates Foundation, which had supported RHSC through its formative years. Since then, a coalition of funders, including DFID, UNFPA, USAID, as well as Gates, have financially supported the RHSC. Keeping the RHSC as several termed it “lean and mean” appears highly prized, particularly by the donors who are providing direct support. Many key informants were pleased and surprised that the Secretariat is able to do so much so well with so few. As one former donor expressed it, “The Coalition: I’m always surprised, it’s like three, four people working out of a closet”. Furthermore, though most donors are in keeping the Secretariat trim, some members expressed need for the Secretariat to add technical depth. Some key informants, including within the Secretariat itself, recommended adding a deputy director. Others suggested that added expertise on policy, advocacy, and country-level programs would be useful. Informed that the operating budget of the RHSC runs about $2.0 million per year and asked whether they thought this good value for this level of investment, a number of key informants also seemed surprised that such presence as the RHSC has could be based on that budget size. The arrangement with PATH enables some specialist functions to be bought in fractionally such as budget administration and human resource management. Some time is regularly incurred (e.g., about 50% of a project administrator) and other services from PATH are called on as needed in a flexible cost effective arrangement. Outsourcing is also used when needed, e.g. website maintenance. Consultants are used selectively for largely strategic pieces of work and this is appropriate. These various arrangements appear to have provided the Coalition good value for the money that donors have invested. 2. How able or nimble is the RHSC to pursue new opportunities and accommodate new ideas and new areas of focus (as in the work, for example, of the Secretariat and Working Groups)? Recent Executive Committee Chairs have encouraged the Secretariat to exercise leadership and move in new directions, at times ahead of what some members would like. The two most outstanding examples are the connections made with manufactures and the Hand-to-Hand Campaign. Successive Chairs supported and encouraged the director to move into these arenas. Manufacturers have brought new perspectives and understandings of the actual total market and have joined the RHSC in significant numbers. Hand-to-Hand has potential to be the starting point for a broader effort to reinvigorate the entire field of FP/RH, notably including commodity security, with new funding streams. 93 Evaluation of the Reproductive Health Supplies Coalition The RHSC Director had high praise for the successive Chairs who have presided over the Executive Committee: “The Coalition Chairs have been as different as day and night and yet each one was absolutely perfect for the time. Who knows what it would have been like if they were different? But they were perfect.” However, he has also been able to move with alacrity to take advantage of promising openings and opportunities. While donors are also members of the Coalition, does this have a downside risk? Being too far in front of members or being seen as too donor-driven could dampen the energy and enthusiasm upon which the Coalition ultimately depends. As experiences with the manufacturers and the Caucus indicate, the Secretariat has moved effectively to accommodate new ideas, those encouraged by donors as well as those arising from members. Some of this has also occurred as a result of Innovation Fund support for initial studies. For example on pre-qualification and online procurement, seed funds led to larger well- funded programs implemented respectively by Concept Foundation and UNFPA (AccessRH). These emerged from “ideas discussed amongst a group of people, perhaps moved forward by one or two organizations that emerged into the large scale strategies that would evolve into programs that should have impact”. 3. To what extent are RHSC structures and processes appropriate for its mission? How well has the RHSC worked to ensure efficient and effective working by its working mechanisms, individually and collectively – considering their respective roles, structures, processes, modalities of working, and working relationships to each other? • How effectively have the Working Groups, Regional Fora, and Caucuses “connected” their work plans to each other, seeking and achieving synergies in work plan objectives, priorities, and activities? The mission of the RHSC is “to ensure that every person is able to obtain and use RH supplies”. (TOR, 2009, p.3) Towards achieving this vision, the RHSC presently encompasses more than two hundred organizations as Coalition members. The structure established within which members work together, as depicted in the 2009 Terms of Reference (p.5), is: 94 Evaluation of the Reproductive Health Supplies Coalition The three Working Groups have been idiosyncratic in their operations and variable in their effectiveness. Each has had its own ebbs and flows over the years. Each has developed its own unique character of participants. The SSWG continues as the work space for the original core of logistics experts, largely influenced by the USAID Deliver Project; MDA has been the most explorative of the groups, reaching out most recently to learn about promising new products and possibly influencing how they may come to market; and the RMA has been consistently most concerned with moving beyond commodities to change policies and have an impact at country level. The engine that drives Working Groups towards effective performance is the combination of funding combined with strong leadership. The SSWG provides a good example of both being in place. The Innovation Fund provided support for some development and testing of particular tools. However, SSWG also benefitted from many participants being a part of the Deliver Project. That the USAID COTR of Deliver has also been an active participant in SSWG, serving for a time as its head, also had distinct advantages. The Working Groups were originally conceived (and continue to be perceived by many) as the engines that drive the RHSC. This structure however has largely been outmoded by two factors: 1) rapid, substantial expansion of the RHSC membership, particularly in the global South and among such non-traditional partners as manufacturers, and 2) the effectiveness of Secretariat leadership in occasionally acting, especially in recent years, ahead of and at a level above a consensus among members. On the one hand, country-level involvement in, let alone country ownership of, the Coalition has not yet been fully embraced or even partially realized. On the other, leading from behind has sometimes been supplanted by leadership from the top. To accommodate such developments moving forward, structural adjustments will be needed. This will become particularly evident as the Coalition moves further along the supply chain towards country level actors and beneficiaries. A question for further consideration is as follows: Does the structure of Working Groups, comprised mostly of high level experts addressing global issues best serve the needs of an organization with the following vision: “All people in low and middle-income countries can access and use affordable, high-quality supplies, including a broad choice of contraceptives, to ensure their better reproductive health”. Apart from major structural shifts, some adjustments and improvements have already been made and others are needed. Stagnation within the Working Groups has been met through leadership transitions which are ongoing in all three. In addition, the Secretariat has encouraged that incoming WG heads undertake visioning exercises within their Groups. Attentive to the risk and actuality that Working Groups become staid over time, the Secretariat has also changed how it operates in relation to this structure: assigned staff have become somewhat more directive in providing support and oversight. Albeit within a particular area of interest, the members themselves sought to address a different need – lack of connectivity among the Working Groups. Within the Caucus, formed to give attention to new and underused commodities, Caucus members sought to consciously connect with and implant ideas within the three Working Groups. Results have been mixed, and the need for better connectivity remains. To an extent Secretariat staff supporting the different Groups can become the connective tissue among the three. This should extend as well to the new structures emerging within the RHSC, particularly the regional forums. For example, a donor participating in an MDA meeting a few years ago, held in conjunction with an annual RHSC meeting, noted that those from Latin America had a lot to offer on total market approaches. The MDAWG leader teamed up with the Secretariat staff member supporting the 95 Evaluation of the Reproductive Health Supplies Coalition LAC Forum to conduct MDA/LAC Forum meetings. Some efforts have been made and Spanish speakers from both groups have met by teleconference to discuss market-related developments. This evidence that LAC members have been involved with MDA is promising. If the purpose of the RHSC is to impact knowledge and understanding about FP/RH commodities at high levels, support and visibility that RHSC gives to AccessRH and RHI has had some success, though there remains continuing need for improvement. However, if the country-level influence is a serious concern of RHSC, the Working Groups have not had nor may they be able to have much impact. Given almost complete lack of direct connection between Working Groups and country-level members – only one in-country resident appeared to be represented within the three Working Groups – it will be appropriate to determine if the Working Groups as structured are optimal for the future of the Coalition. 4. How effective have strategic and work plan decision making approaches used by the RHSC and its components been (Executive Committee, Secretariat, Working Groups, etc.)? To what extent are decisions – for example, what is in the RHSC strategic plan and working group work plans – driven by country needs and priorities versus global agendas? How well are constituencies (e.g., developing countries, civil society, private industry) represented in decision-making structures? Though the evaluators did not participate in an Executive Committee meeting until the end of this assignment, they interviewed several past and present Executive Committee members, including Chairpersons, and reviewed a number of relevant documents. To a more limited extent, the evaluators participated in Working Group conference calls and meetings. Key informants throughout the membership were also queried about the effectiveness of the RHSC and its component structures. The evaluators draw on this comprehensive information base to answer the above question. The Executive Committee provides governing oversight for the RHSC. As such it is comprised of representatives of those organizations which provide funding to the RHSC, as well as those from other established organizations representing various and different aspects among other members. For example, the International Planned Parenthood Federation has sat on the Executive Committee in recognition of and to represent the country-level breadth of the Federation. Also on the Executive Committee are representatives from the Pan American Health Organization, the West Africa Health Organization, and the World Bank. The heads of the three Working Groups are also on the Executive Committee. The Executive Committee meets bi-annually to review previous accomplishments, assess the current situation and opportunities, and put forth ideas and directions for activities and accomplishments in the coming year. The latter are formalized as an annual SOW to guide the RHSC Secretariat. There is clearly an inner core within the Executive Committee comprised of the four donors supporting the Coalition. Many key informants, who had participated in Executive Committee meetings including those from donor organizations, commented on this phenomenon. While understandable, caution is needed to ensure that this membership organization does not get driven too far in different directions by donor prerogatives. Similar balance and risk assessment are needed on the part of the Chairperson. In the past this has been a donor representative, albeit not always a direct funder of the RHSC. The present Chair is from DFID and received high praise from a variety of key informants for the impact she is having upon RHSC. 96 Evaluation of the Reproductive Health Supplies Coalition Nevertheless, vigilance is needed to maintain balance between this donor subgroup and other members of the Executive Committee. It is noteworthy that the present day Coalition originated from interest among a group of donors to collaborate to raise the profile of FP/RH commodity security. While some perceive that this collaboration continues, others thought that more concerted effort should be given, under the rubric of the RHSC, to fostering more donor collaboration. One key informant suggested that a Donor Working Group could be a useful adjunct to the current structure of the RHSC. Other key informants suggested updating the Executive Committee so that it could better represent the fuller membership. For example, one asked “Why not have a manufacture (representative) on a rolling basis be a member of that Committee?” Another possibility would be instead of PAHO and WAHO as proxy seats, more faithfully represent country level by having dedicated seats for in-country organizations. A highly placed respondent within the RHSC suggested adding to the Executive Committee seats for qualified and experienced technical experts, such as Steve Sinding. Beyond the Executive Committee, the structure of the RHSC Secretariat supporting Working Groups has worked well. While each of the three Working Groups has experienced its own ebb and flow, all three have been productive in terms of hosting collaborative discussions among and across participating member organizations, generating work streams, and undertaking activities consistent with each Working Group’s work plan and the overall RHSC strategy. The three Working Groups clearly well represented RHSC members in the early initial years of the RHSC. However, membership growth since then, and particularly in the most recent years, has resulted in the Working Groups becoming more an elite sub-culture of founding members than a vital and vibrant part of expanding interests and possibilities. Many of the newer constituencies that have joined the Coalition, notably country-level organizations and manufacturers, are not present in Working Group discussions or meetings. There is clear interest in the Working Groups in having more input from the Global South. However, except for the recently appointed co-Chair of RMA, no country-level residents regularly participate. Similarly, involving manufacturers, and including those producing generic FP/RH products, would provide a much needed added dimension. Perhaps different structures or means for engagement are needed to bring in these new and different perspectives. Exceptional are the annual meetings which also serve as times for the Working Groups to meet. Several key informants report that this has not been a satisfactory solution for gaining broader participation. A great deal of what occurs during Working Group meetings held in conjunction with annual RHSC meetings is taken up with bringing those not closely involved up to speed. As one Working Group head observed about the meeting in Addis: “When the Working Groups had their meetings they were, I would say, half and half, people that had participated before and wanted to talk about the work that was underway and what was getting done and the other half of the people had no idea of what was happening but they came because they were there for the meeting at the end of the week, you know? So I think that was not a really successful endeavour. We probably should not have had Working Group meetings because I think the Working Groups weren’t able to do anything.” This is not to say that the annual Coalition meetings have not themselves provided a very useful structure through which the RHSC builds understanding, ownership, and collaborative possibilities. As two key informants from European donors related: 97 Evaluation of the Reproductive Health Supplies Coalition “Annual meetings are an important platform for exchange. This is very equal.” “There is shared understanding of importance. Shared enthusiasm – donors, people in country all keen to work together. The RHSC can channel this.” The structure of an Executive Committee, a Secretariat, and three precisely delineated Working Groups put forth in in the TOR continues to support RHSC’s ongoing operations. The principles outlined in the TOR have also guided the RHSC in subsequent years towards working at different levels and for distinct aims. How these have evolved provides a useful outline for and insights into what the RHSC has become. Flexibility in accommodating new structures that have emerged from member interest, such as the Caucus and the regional forums, has been a helpful adjustment. However, those at country level, at the farthest reaches but closest to where solutions lie, remain to date members more in name than actual involvement and participation. Some exceptions on RHSC involvement at country level are both interesting and instructive. In Uganda and Ethiopia, which were purposefully selected as sites for annual meetings, RHSC engaged directly with national governments in both countries as part of the process. In both countries, positive outcomes for FP/RH commodity security occurred. While annual meetings are not a format for fostering country involvement on a routine basis, these experiences do show that it can be done and done effectively. Similarly, in the small number of countries where RMA Working Group efforts have reached to country level is there some blueprint for future RHSC activity. Country ownership, clearly a long-term aspiration, may helpfully and hopefully come to the fore when the RHSC re-calibrates its strategic compass. 5. How effective is the partner-centric model for the RHSC? How effective has the RHSC been at catalyzing partner investments and leveraging partner resources (funds, people, etc.) not only for RHSC-inspired initiatives or the RHSC itself, but for RH commodity security in general? • In particular, to what extent has the RHSC’s partner-centric approach to the Working Groups, Regional Fora, and caucuses, and implementation of their work plans, been effective (e.g., dependence on partners for working group coordination and funding of work plan activities)? This “coalition of the willing” is clearly buttressed by some of the Working Group members having separate and closely related project funding, such as the Deliver and Advance Family Planning Projects. Alternatively, the Coalition of the willing falters when resources are not available to participants. As an example of this, Project RMA actually had somewhat of a deleterious effect on the functionality of the RMA Working Group because only three of its members were financial recipients of this Project. In this context, the Innovation Fund has been a very useful addition of funding. However, as noted by the members themselves in responding to an internal review of the Innovation Fund, the criteria were not always clear, the projects and studies which resulted not always strategic or cutting edge, and administration and follow-up not what they should be. Similarly unclear, or at least not sufficiently transparent to the evaluation team, is the way in which the RHSC attributes member activities as RHSC leveraged funding. Though it is common practice as one key informant noted to “count everything in sight”, the Director states that the RHSC was conservative, even scrupulous, in how it calculates leveraged funding. Leveraging should be more on a sliding scale than one of cut-and-dry attribution. Thus the RHSC had earlier roles in supporting some Coalition activities than others. For example, Innovation Fund 98 Evaluation of the Reproductive Health Supplies Coalition support for early work on pre-qualification was instrumental in attracting much larger funding to continue developing this approach. For other Coalition efforts, such as RHInterchange and CARhs, RHSC had relatively late involvement and has more a promulgating than developmental role. As the Director recalled, correspondingly less or no amounts were counted as leveraged on account of such initiatives. Some organizations within the Coalition are clearly more advantaged and connected with FP/RH commodities and supply chains. For example, staff members from the Deliver Project have had continuously significant involvement within the SSWG. This has occurred in part because USAID has made long-term investments to support this capacity for strengthening contraceptive supply systems at JSI. Nevertheless, a variety of other organization less advantaged and not so directly connected with logistics and supplies have become members of RHSC in recent years. It is a credit to organizations such as JSI that staff involved in the Coalition look favorably upon this expansion. The recent SSWG meeting discussed not only additional improvements that can be made in supply systems strengthening but also how to more fully involve and engage with Global South members. 6. Are there adequate systems in place to monitor and report RHSC performance? There are neither adequate systems in place to monitor and report against Goals One and Two nor is the MEF measuring the right things. The strategic plan of 2007 contained a list of 58 illustrative indicators to suggest the kinds of information that could be used to monitor progress of the 9 objectives under the three goals. A monitoring and evaluation framework (MEF) was then developed, refining a list of 30 specific, operationalized and collectable indicators (see Annex 7). These indicators had baseline values (mostly set for 2007), a strategy for collection, and a means of verification. It was also agreed that indicators should already be collected by one or more Coalition partners (or agency of a partner) and they should be relevant to 14 focus countries receiving targeted support by the Reproductive Health Interchange (RHI), Project RMA (Resource Mobilization and Awareness), and the Global Programme of the United Nations Population Fund (UNFPA). The intention was to review the Strategic Plan and M&E in 2009 at the end of a grant from BMFG, and then again in 2015 to coincide with the end of the Strategic Plan. At the request of the evaluators, in 2012 the Secretariat kindly prepared another monitoring report (see Annex 13). Data for recent update of M&E indicators only gathered after the evaluators specifically requested it – strong indication that it is neither highly valued nor useful for monitoring and measuring the activities that the Coalition is undertaking and the progress it is making. Only the indicators under Goal Three actually relate to what the Coalition contributes and these are provided as illustrative examples. The evaluators found that the monthly newsletter is a better place than the MEF to regularly monitor RHSC activities and accomplishments. The M&E indicators under RHSC Goal Three that report on what RHSC itself does do not appear to be used to monitor progress. 7. How has the Secretariat’s role evolved over time? Is there evidence that it has assumed greater responsibility for achieving RHSC objectives? What are the Secretariat’s most essential contributions? To what extent is the Secretariat a support to, versus substitute for, 99 Evaluation of the Reproductive Health Supplies Coalition the work of other RHSC working mechanisms and RHSC partners? 9 Has the Secretariat enabled the RHSC to accomplish more than it could without the Secretariat? How is it decided whether an activity is the responsibility of the Secretariat or other RHSC working mechanisms or RHSC partners? Leadership of and within the Secretariat is central and key to the success it has had. In the current Director, the RHSC has an experienced professional who has been able to balance daily managerial responsibilities for keeping the enterprise on course with advancing toward new opportunities that may be on or just over the horizon. As one senior level key informant, knowledgeable but only occasionally and directly in contact with the RHSC, has noted: “The director, John Skibiak is a strong person…he came to the job knowing a fair amount about contraceptive supplies, use and affordability…and since he’s been on the job he has learned a lot, increased his networks, and he has done, what seems from my position a very good job”. Another offered this perspective: “John Skibiak…is a very good person to (head the RHSC) because he’s broad and can work with a large set of people in a way that is helpful for bringing in different perspectives. And from another: “John that's been the driving force behind (raising the prominence of FP/RH commodity security)…he's got involved in a whole range of forums which we never used to get involved in. But the problem is he has a limitation – he’s one man!” The evaluators will not see the Executive Committee in action until the end point of this assignment. They have however interviewed past as well as present Chairpersons as well as a number of Executive Committee members, and are able to draw some conclusions on that basis. Particularly helpful have been perspectives provided on the role of the Chair and how that has changed and contributed. The Director himself credits the leadership provided by successive Chairpersons “the Coalition Chairs have been as different as day and night and yet each one was absolutely perfect for the time.” Another key informant who has participated in a number of Executive Committee meetings offered this similar reflection: “The role of Chair is incredibly important… I think Julia has brought an enormous amount of enthusiasm, a sense of can do….There’s no doubt about that, but other Chairs have brought equal skills but in slightly different areas, … I think each has been the right Chair at the right time and that’s been very, very important … the Executive Committee is one where you can have a free and frank discussion…My one misgiving, if you will, is there a subset of them what’s got the power.” The Executive Committee, with the caveat that there is a subgroup of donors within it that holds greater sway, appears to have functioned well through the years. Donors, in the person of the Chair, have encouraged the Secretariat to take the lead for some initiatives, such as relations with manufacturers and more recently the Hand-to-Hand Campaign. The RHSC is also currently serving donor interests in preparations leading up to the FP Summit. While these may all be to good effect, such top-down leadership can create separation from members or at the least not make use of members in areas where they have expertise. _________________________ 9 The working mechanisms of the RHSC are the Chair and Executive Committee, Executive Director and Secretariat, Working Groups, Regional Forums, and Caucuses. 100 Evaluation of the Reproductive Health Supplies Coalition The Executive Committee, Secretariat, and Working Groups have remained largely intact, with the regional Forums, Caucus, and Task Force useful additions along the way. This is not to say that the overall structure is completely in balance with what the RHSC has become. For example, the Secretariat staff work long hours, as a number of key informants who are in regular contact and receive routine support have noted. In addition, the established structures within which the RHSC operates do not and may not be able to encompass the membership as it has grown in recent years. The Secretariat is clearly committed to serving the Working Groups and other structures. Support provided by the three Secretariat staff to these various structures is well appreciated and seems vital to maintenance of structural operations. However, there is a price to pay for running an organization with such extensive reach with so few permanent fulltime staff. In the whirlwind of daily activities and ongoing responsibilities, strategic oversight and forethought is quick to fall by the wayside. There has at times also been staff disaffection and turnover, perhaps having to do more with mismatches between the level of work required to serve what members need than with direction and guidance being provided. Staff departures led to less technical expertise within the Secretariat, which several key informants noted as a loss. Remarking on staffing needs within the Secretariat, many key informants, and particularly those most involved in Working Groups, responded that higher levels of Secretariat-level expertise would be useful. Several respondents noted the lack of depth in certain areas such as policy, advocacy, and country-level experience among Secretariat staff. Secretariat staff themselves noted need for more expertise and experience. The director and staff alike would like to see a deputy director level position which could both share the workload and deepen available expertise. 8. How effective has the Innovation Fund been in supporting working group work plans, catalyzing significant work that would not otherwise have happened and fostering innovation? The Innovation Fund began in 2008 with support from the Gates Foundation, which allocated over $2.0 million to support RHSC Working Group generated studies and projects. To date, the review committee (comprised of selected Executive Committee, Secretariat, and PATH staff members) has approved 16 awards. With funding in the range of $50,000 to $200,000, successful applications from the three Working Groups included studies of new areas for commodity security advancement, such as pre-qualification; applications of specific logistics and market analysis tools within country settings, such as application of the Market Segmentation Tool in Honduras and Madagascar; and production of a documentary film, Empty Handed, on the importance of contraceptive logistics. A 2011 internal review of the Innovation Fund solicited input by surveying members. The evaluators agree with the main conclusions of this review: lack of clarity about the RHSC strategy and how this aligns with the Innovation Fund; that what was funded was often not innovative and sometimes self-serving of organizational rather than field interests; and that viable country-level proposals were not forthcoming within the construct and processes through which proposals were developed and reviewed. To correct for the latter, remaining funds of some $270,000 have been set aside for country-level applications from civil society organizations. To facilitate this, the RHSC has asked and given a small amount of funding to Population Action International to identify and assist civil society organizations with proposal development. 101 Evaluation of the Reproductive Health Supplies Coalition The Innovation Fund has also fallen short on what could have been a hallmark of these awards: fostering collaboration among member organizations. Almost all awards involved projects and studies implemented by a single organization. Left to their own and despite the connections that occur in the Working Groups, organizations seek funding that is more likely to further their own interests and agendas than to more broadly contribute to the field. Proposal criteria that prioritize connections and collaborations across member organizations should have greater weight. Recommendations about the future of the Innovation Fund were summarized towards the end of the internal review document: “Several ideas for different funding options and additional budgets where put forward these include; a seed fund, an ad hoc fund, a risk budget, a priority fund, a developing countries budget, a capacity building budget, an opportunity budget, a side budget for pressing issues in activities underway. There is a need to be realistic on how much can be expected for relatively small amount of money.” Some of some initiatives supported by the Innovation Fund have been intended to have some impact at country level. The Table in Annex 9 summarises these results for those such initiatives that have been completed. What should happen to the Innovation Fund going forward? Without some funding going for Working Groups, they will very likely lose what impetus they have. Clearly the Innovation Fund has been useful for moving some 16 projects and studies forward that were of interest to Working Group members. However, for the most part these have largely not been innovative nor has the Fund attracted proposals from far beyond the usual suspects. Several key informants felt that the Fund had been misnamed and needs to be renamed. With a different or modified structure to promote a different level of involvement and interest, Country Catalyst Fund may be a more appropriately named fund and a new direction. 9. With specific reference to operations of the RHSC Secretariat: • How effective has internal management of the Secretariat been? For example, managing human resources and the relationship with PATH. • Is the Secretariat staffing of the right size and skills mix? Has the Secretariat been able to recruit and retain the kinds of people needed? If not, why? • Are there appropriate controls and oversight for Secretariat finances, and are they exercised appropriately by the appropriate entities? • What have been the impacts, positive or negative, of the decision that the RHSC not be a legal entity and that it is in effect embedded within another organizational structure (PATH’s)? What are the costs and benefits of this lack of autonomy? • Is the Secretariat’s location in Brussels the right balance between costs and benefits, relative to other possible locations, including in a developing country? • The RHSC Secretariat outsources certain core functions (e.g., website maintenance) and ad hoc studies/assessments (e.g., communications assessment). How efficient and effective has this approach been? Management within the Secretariat has been effective, though not without occasional problems. Recently, during a period of rapid growth and perhaps corresponding high pressure, some staff changes were effected. As happens in most organizations, these did not take place without tension and disruption. However, the transitions have occurred without any lasting deleterious 102 Evaluation of the Reproductive Health Supplies Coalition effect on remaining staff. The greater risk may well have been to not make changes leaving some dissension within the ranks to fester and foster discord. What was lost in the process was some depth of experience and expertise from one of the staff who left. Though this particular skill set will not be replaced, the Secretariat could benefit from having, in addition to the Director, at least one other more senior staff member. PATH which serves as the fiscal agent for the RHSC was instrumental and reportedly very helpful in remedying these internal staff problems. Having a full-fledged Human Resources component of PATH to draw on was timely and helpful. The arrangement with PATH also enables some specialist functions to be bought in on a part time or as needed basis. This appears to be a flexible cost-effective arrangement. Outsourcing is used appropriately in such areas as website maintenance. Consultants are used selectively for largely strategic and/or time sensitive pieces of work as appropriate. Input from PATH enables the Secretariat to make best use of the donor funds at its disposal. The four donors all have different rules about overheads, procurement, staff costs, etc., and the funds have flowed at varying intervals over different periods. The project administrator at PATH is skilled in matching the needs of the secretariat with the funding options available within the entire operating budget, giving good value and relieving secretariat staff of this tricky process. Brussels is an expensive place to be based (particularly for staffing costs), but it is difficult to do a general comparison with other locations. It cannot automatically be assumed that relocation to somewhere else might save money overall. Less expensive staff in Africa might have to be housed in more expensive offices and effective communications may be more costly. If the re- location of the Secretariat was deemed to be necessary consideration would have to be given to the value for money of other locations. The Brussels location though not particularly advantageous now has the advantage of being well established as the home office of the RHSC. In addition to donor funding, the Secretariat has sometimes mobilised contributions from partners to support ad hoc activities carried out by secretariat staff. These are usually in the form of support for travel to events, meetings etc. Often these are given directly to avoid an overhead being charged and pushing up the overall cost. PATH charges an overhead for its services which is in line with its donor rules. On only one occasion during the course of interviewing more than 60 key informants were some lingering concerns raised about conflict of interest: PATH managing an organization working within an area where PATH itself would be able to compete for business and funds. However, even in this one instance, this was raised in relation to a perceived incident that had occurred many years in the past. PATH and the Secretariat expressed sensitivity to the potential of conflict-of-interest, but that this is being well managed. The evaluators did not learn of anything to the contrary except for this one reported instance. 10. How effective has the Executive Committee been as a governing body? Is its involvement in Coalition affairs of the right kind and at the right level? A number of active and former Executive Committee members were interviewed as key informants for this evaluation. Though the evaluation team will not have an opportunity to observe the Executive Committee in action until the end of this evaluation, it appears to have served the RHSC well. However, what was needed in the past may not be what is needed at present or for the future. 103 Evaluation of the Reproductive Health Supplies Coalition The Executive Committee is the governing board of the RHSC. However, it is a committee rather than a board because the RHSC is not an officially registered organization. The Executive Committee includes all four funders of the RHSC, heads of the three Working Groups, and representatives of organizations with notable presence in international commodity security and family planning, namely the International Planned Parenthood Federation (IPPF), the Pan American and West African Health Organizations (PAHO and WAHO), and the World Bank. The Executive Committee meets bi-annually. During these meetings to review accomplishments, assess the present situation, and set future directions for the coming year. Considerable attention has been given in the past to what the Working Groups have been doing and what each plans to do in the coming years. One participant noted that this dynamic seemed to disadvantage the Working Group heads from engaging in full participation as Executive Committee members. Executive Committee members, including as observed by the funders themselves, noted that the donor representatives form an inner core. Accordingly and almost as a given prerogative, the funders exercise greater influence within the Executive Committee. Chairpersons, also from the donor community though not necessarily direct RHSC funders, have been particularly influential in shaping the RHSC. Particularly close working relationships have often developed between the Chair and the RHSC Director. Though there is some risk that this could lead to becoming a donor-driven entity, to date this does not yet seem to be a concern among members. Key informants at various levels encouraged expanded representation on the Executive Committee. Among specific recommendations offered were to include new categories representing member constituencies, such as manufactures and country-level. In addition, one well-placed key informant suggested deepening discussions and deliberations within the Executive Committee by adding independent experts on commodity security and international family planning. 11. Have the Coalition’s relationships with funders of the RHSC worked to best advantage? For example, have disbursements been timely and funding agreements not overly dictating in how funds can used or burdensome in their reporting requirements? The direct donors of RHSC, as widely noted by Executive Committee members, and notably donor representatives themselves, form an inner core of the Executive Committee. To some extent, donor presence and participation on the Executive Committee may usurp leadership from within the RHSC and dampen fuller participation by other Executive Committee members. One corrective for this within Board sessions themselves would be to add independent technical positions to the Executive Committee and reserve some rotating seats for constituencies not yet well represented on the Executive Committee, such as manufacturers and country-level representatives. Another solution could be to have donors also rotate among fewer seats, while at the same time finding other modes to facilitate their input and promote inter-donor collaboration on FP/RH commodity security. 104 Evaluation of the Reproductive Health Supplies Coalition The Board Chair, who has always been a funder, albeit sometimes not a direct donor to the Coalition, has usually exercised considerable influence over RHSC, as may well be appropriate. However, there are two related risks: RHSC directions may extend beyond where members wish their organization to go and the RHSC and its Director come to be viewed as donor-driven by the members. The evaluators had no input that this was becoming a prevailing perspective among members. However, it is a risk to be considered. Sometimes to move quickly toward a particularly attractive goal is to leave followers and strategies far behind. The donor prerogative to give as well as withdraw funding is a powerful determinant of direction. With Gates Foundation support, the RHSC was able to successfully move the Pledge Guarantee for Health (PGH) forward as a promising approach to smooth funding cycles and commitments which can be disruptive to procurement. However, slowness in moving PGH forward led to withdrawal of operating funds. As one key informant stated, the PGH “was not the silver bullet hoped for”. Should other funders come to the fore, this interesting approach may yet find opportunity to help solve a perennial procurement problem. Donor disbursements have sometimes caused complications in allocating expenses to particular donations. However, the support provided by PATH appears to have managed this well. If a different funding arrangement were to be used at some point, there would likely be some additions in costs and losses in efficiencies. C. Future Directions 1. In light of the current and emerging environment, are the RHSC’s strategic objectives and operational strategies the right ones? Is the Monitoring and Evaluation Framework measuring the right things? What changes are needed, if any, to these and to the Strategic Plan to maintain and strengthen the RHSC’s relevance and comparative advantage/value- added, and expand its impact at all levels? Strategic Plan goals and objectives Goal 1: Increase availability, predictability, and sustainability of financing for RH Supplies. Objective 1.A: Increase flow of new resources. Objective 1.B: Ensure that countries and their governments make use of the resources already at their disposal. Objective 1.C: Mobilize Total Market Resources Goal 2: Strengthen capacity of health systems to deliver RH supplies in a sustainable manner Objective 2.A: Ensure the integrity of systems for supply chain management, both globally and at country level. Objective 2.B: Provide a coordinated, short-term response to stock outs, supply disruptions and other failures that occasionally befall functional supply chains. Objective 2.C: Promote effective delivery of RH health supplies in settings where routine systems for supply chain management have broken down or ceased functioning. Goal 3: Assure the added value of the Coalition as a productive and sustainable global partnership through efficiency, advocacy and innovation Objective 3A: Foster greater harmonization among stakeholders at global and country level to generate economies of scale, maximize impacts, and minimize the risks of redundancy and confusion. Objective 3B: Generate, disseminate, and facilitate the application of knowledge through innovation, research, and documentation. Objective 3C: Offer a credible and sustainable institutional framework within which stakeholders can collaborate, build commitment to supply security, formulate strategies for change, develop common indicators of success, and measure progress against them. 105 Evaluation of the Reproductive Health Supplies Coalition • More specifically, what areas of activity should be considered for addition to Secretariat and/or working group work plans, in order to support and monitor countries to strengthen supply chains, build capacity for advocacy, and other that will further RH commodity security? Are there additional working groups to recommend? The 2007-2015 Strategy provides a useful historical marker for assessing Coalition ambitions and accomplishments over the last five years. However, as the RHSC Director stated, this strategy is more “aspirational” than an up-to-date representation of what the Coalition was at the time. Furthermore, the strategy document is itself not readily absorbed or actuated by members. As early as 2008, respondents to the members’ survey expressed lack of clarity. This was echoed in the sustainability study conducted in the same year. To make the strategy more understandable, a three by three matrix was developed and placed on the RHSC website (see (http://www.rhsupplies.org/about-us/our-strategy.html). The Strategic Plan was developed by an external consultant well known to many in RHSC who led a consensus process. Though one Executive Committee member recalled being personally opposed to developing such a strategy, the final document was reviewed and endorsed by the membership as a whole. However, with the passage of time it has not proven a reliable guide or map for developing measures to assess progress along the way. The Monitoring and Evaluation Framework of 2009 posits indicators in line with the Strategic Plan. However, with the exception of illustrative examples to track progress towards the third objective, these indicators do not usefully inform the Coalition on its own activities. Therefore, in broad strokes the M&E framework neither monitors nor measures markers of Coalition progress. Perhaps as a result, performance that accords with the M&E is not kept up-to-date. In the most recent annual report, for example, only indicators for Objective three, those pertaining directly to activities and accomplishments directly within the purview of the RHSC Secretariat, were recorded and reported. At the request of the evaluators, RHSC completed an update in early May. Nor is updated tracking of indicators, in the form of a dashboard or other similar format, a part of the RHSC website. However, some key informants did not find the current website useful or up-to-date. For example, one donor staff member reported that for her it was “not a go-to site, not specific enough or with up-to-date information at country level.” These shortcomings are clearly recognized by the Secretariat. Ongoing recruitment for an M&E specialist is anticipated as a solution. However, the evaluation team holds that the problem lies not with how frequently indicators are being tracked and measured, but what are the actual measures that could guide the RHSC. Thus, not only the M&E needs revision but the strategy as well. A significant shortcoming in how the RHSC is anchored and functioning is its lack of a theory of change. When questioned about the existence of a theory of change, the Director of RHSC indicated that the three-by-three interactive matrix in its website was the best representation they had to engage members in understanding the various components and operations of the Coalition. This shortcoming needs to be addressed. The evaluation team posits a process that also includes revisiting and recasting the RHSC’s Strategic Plan. A theory of change that accurately captures what the Coalition is today would be a great starting place for determining what the RHSC could become in the future. As important as what may be captured in such a theory of change would be the process by which it is put together. This may go a long way toward defining what the RHSC has become and how it will continue into the future. 106 Evaluation of the Reproductive Health Supplies Coalition Beyond these inadequacies in indicators that track what the Coalition contributes and accomplishes, the RHSC does not have a clear, concise and updated articulation of what it ultimately expects to accomplish and how it proposes to do that. It will clearly need to do so as part of its upcoming funds to continue operations. Within the Working Groups, the evaluators would suggest that more attention be given to involving country-level participants, including field-based staff of the large INGOs. More attention to connecting with country-level implementation is also advisable. Among the three Working Groups, only RMA gives consistent attention to this at present. However, it may well be that new structures are needed to more fully involve and move toward country-level implementation. As could be occurring within the LAC forum, movement toward country-level partners and implementation will not happen automatically. It needs to be nurtured and encouraged by the RHSC. Strategy revision to reflect that would be a good starting place. In addition, forming new Working Groups, which is already happening within the LAC Forum, could be useful structures to consider. Special attention may be needed to ensure that such Working Groups represent and function at country or sub-regional levels. The RHSC has within its core statements about its mission and vision the idea of country orientation if not country ownership. The Coalition has also encouraged and to some extent succeeded in expanding membership to country level. Key informants were clearly supportive of the idea that the RHSC be more focused upon the country level. While there was some consensus around maintaining a core focus on FP/RH commodities, there was also unanimous support for doing more to involve and within countries. As has also been affirmed by the entire field of FP/RH commodities, security means all the way along the supply chain, including to the last mile where products are accessed and methods used. There is interest within the Working Groups in having more input from the Global South. However, except for the recently appointed co-Chair of RMAWG, no country-level residents regularly participate. Other constituencies that have joined the Coalition, notably manufacturers, are similarly not present in Working Group discussions or meetings. Exceptional are the annual meetings which also serve as times for the Working Groups to meet. However, as several key informants report, this has not been a satisfactory solution for gaining broader participation on a regular basis. Bringing two regional forums under the umbrella of the RHSC is a move in the right direction. However, as experience with other regional entities indicates, such regional organizations may be too large to translate into effective activity or results at the country level. Sub-regional groupings may be needed, perhaps even pairings between advancing countries (Ethiopia, Rwanda, and Malawi) and countries with near-term potential to advance (Tanzania, Ghana, Zambia). To encourage and involve countries directly and continuously with the Coalition may require formation of new working groups. Movement in this direction will require different and more staff as well as funding to support activities at country level. 2. How can the RHSC broaden the numbers and kinds of organizations, particularly from the global south, that are involved in working group work plans? For example, how can the Innovation Fund, or a mechanism like it, be used to this end? Are there are other means that might be pursued? If the RHSC were to move more completely toward its principle of “country ownership”, it is the countries that should become the engines of the Coalition rather than the Working Groups. Though the Working Groups, particularly as currently formulated, have successfully served as 107 Evaluation of the Reproductive Health Supplies Coalition the “technical engines” to drive the RHSC, alternative approaches can be considered. As raised by one donor, “Why don’t we have country-level impact? We still have that study from way back when, Mercer, that showed end-to-end supply chain: where are the problems that ultimately manifest themselves in stock-outs? If you attack the supply chain at country level, you make about, I think they said, 70% difference. If you attack the problem at higher, global level, which is where we focused: about 30% difference. Let’s be honest, that’s what we did – Pledge Guarantee, Access RH CAHRs. Fine! Done that! Let’s get to the 70% now, and that would require turning the Coalition on its head!” The Working Groups as currently formulated are an intellectual top-down approach. The countries as engine would be based on the principle of country ownership of the problems they are trying to solve. However, though many members expressed need for the RHSC to do more at country level, they were also cautious about what and how much the RHSC could actually take on. A number of key informants suggested that the RHSC might consider selecting a small number of countries on which to concentrate such efforts. This would also give opportunity to reconsider the 14 focus countries that the RHSC identified some years ago, more as priorities for donor investments and tracking data than of current interest to the RHSC and many of its members. What should happen to the Innovation Fund going forward? Without a replenished Fund, Working Groups could well lose some momentum. However, as several key informants observed, for the most part awards have not been particularly innovative and the Fund is in need of a new name. The Innovation Fund has already supported 16 projects and studies of interest to Working Group members and is currently being replicated within the LAC Forum. The small amount of Fund remaining is currently being directed towards in-country civil society organizations, which is a useful effort to support country participation. Given the focus on advocacy in this new effort, there is some concern within the Secretariat that only civil society organizations will end up being funded. Country-level needs are important as is fuller commitment to solving supply chain and other problems within the countries where they are occurring. As a respondent to the member survey remarked, “(Being a Coalition member provided) opportunity to compete for funding though we never got any, but is another chance to learn from our shortcomings, as we look for funds to implement our activities“. Perhaps a Country Catalyst Fund, with weight placed on collaborative work at country level, would be a useful adjustment. The country-level presence of the RHSC has been largely untapped to date. Not only have many new members from the Global South joined the Coalition, but many of the long-time INGO members have extensive numbers of staff in some of those same countries. This is a resource to be mined, especially if the RHSC is able to orient effort to working within some select number of priority countries. 3. What measures could improve the performance and efficiency of the RHSC’s components and how they relate to each other, and of RHSC processes (e.g., communication pathways within the RHSC structure, decision making by the Executive Committee)? The RHSC components, as illustrated in the diagram under SOW Question B-3, are structurally connected. However, the Coalition has functioned as a horizontal as well as a vertical organization. In some ways, when it is partner-led and within the “logic of leading from behind” it is organizationally amorphous, loose-knit groups of individuals working on areas of interest and value to their professional lives and development. At other times, the RHSC like other 108 Evaluation of the Reproductive Health Supplies Coalition organizations and particularly where management of funds is involved, is necessarily top-down. In addition, as the Coalition has gained in stature and the Director has matured into his role, leadership in particular directions has occurred apart from member input. As a membership- based organization it is important to maintain this duality of operations. More is needed to foster connections across structures. Annual meetings are one occasion when that occurs, albeit sometimes more in the hallways than in the sessions. However, more could be made of these. Taking lessons from the launch meeting of SECONAF in Addis Ababa (Access for All), where Open Space was used, greater creativity to connect members could be injected into these annual events. This does not mean that less planning is needed. Rather such meetings need to have even better planning, using professional facilitators to help determine which meeting methodologies would be most appropriate to accomplish what. The Caucus for New and Underused Methods has also made interesting efforts to work across and influence work within Working Groups. Given the need for forging connections among RHSC members and structures, perhaps lessons from social network analysis could be applied both to increase connections and to monitor levels of member involvement. 4. The Working Groups, Regional fora, and caucuses, while thought of as the technical engines of the RHSC, are particularly dependent on partner contributions, both in-kind and financial. To what extent does this partner-centric model remain relevant/feasible? Should a different approach be taken to supporting these groups and their work plans that would be more efficient and effective, and yield greater impact? What would be the funding implications of an alternative approach? Dependence on partners has served the RHSC well. However it is also well recognized that the partners upon whom the Coalition has most depended are largely from relatively advantaged and well-funded INGOs in the northern hemisphere. As the membership towards greater Global South representation and focus, will the partner-centric model, in-kind as well as financial, still be viable? Should some different Working Groups be created or emerge to serve need for greater country-level involvement, more Secretariat support as well as some direct funding for participant travel and phone connections may be needed. This would also apply to support ways for engaging more Global South participation in the current RHSC structures. Some key informants reported that the Working Group work plans no longer served well. Others liked both the support provided as well as the relative freedom that different structures have to evolve their own ways of operating. Some key informants thought positively about supplementary funds to support Working Group operations; others appreciated that not having support promoted volunteerism. Just as the RHSC has begun to work with regional forums as adaptations of the Coalition at a different level, so should it support development of coalition-type coordination within countries. This could build on what already exists (through UNFPA-supported Commodity Security Committees and other coordination groups) within a small number of selected countries. While in-kind contributions are as likely (or difficult) to be provided at country level as elsewhere, funding at any level is always a great motivator for participation. Perhaps Global Programme support, reputed to be flexible in supporting contraceptive security improvement at country level, may be tapped. In addition, as mentioned elsewhere, a variant of the Innovation Fund could give special attention to country level projects and activities for improving supply chain efficiencies. The countries as engine would be based on the principle of country ownership of the problems they are trying to solve. 109 Evaluation of the Reproductive Health Supplies Coalition Strategies to increase the participation of Southern members in RHSC structures could include: • Developing different forms of engagement with the current Working Groups. Consultation about new priorities could be done through email or by members reaching out to contact counterparts through their local offices. • Perhaps have one or two country members on the Executive Committee occupy revolving posts. • Forming a country working group which, borrowing from the Caucus practice, informs the work of other Working Groups. • Funding more attendance participation by southern partners at Working Group meetings (i.e., paying for phone calls). • Funding more attendance by southern partners at the Annual Meetings and working out ways to better engage them, and sustain that engagement afterwards. • Paying particular attention to communication strategies to reach existing ‘silent’ Global South. • Developing mechanisms that encourage/promote cross-country exchanges and interactions on commodities to create country level momentum. • Supporting more links between northern and southern organisations, as has occurred in some countries through the RMA Working Group, across the RHSC more broadly, but particularly at country level. 5. As demands grow on the RHSC Secretariat (for example, to support the HandtoHand campaign), to what extent should outsourcing be expanded versus an expanded in-house staffing? What should be the optimal number and type of Secretariat staff? Before the RHSC considers in which circumstances outsourcing would be useful, it needs to take fuller advantage of in-sourcing. Rarely does the Secretariat direct or encourage Working Groups to focus on particular or particularly important areas for inquiry or study. The ethos of being member-led can and could be balanced by a strategic plan that orients Working Groups towards promising lines for advancing FP/RH commodity security. Instead of the current RHSC Strategic Plan that encompasses and coordinates within what member organizations already do, the member-base could be involved in revising the strategy towards higher level goals of what can be collectively accomplished. The Innovation Fund as it has operated to date is a case in point. Leaving criteria for applications wide, the projects and studies subsequently conducted were decidedly mixed. Future funding to support members should have some weighted criteria toward particularly interesting approaches, while at the same time preserving room for others that would emerge from member initiative and creativity. The Hand-to-Hand Campaign provides another example of when in-sourcing could perhaps have been used to the RHSCs advantage. H2H was largely a missed opportunity for getting input from the Coalition member experts in advocacy, especially from the RMA Working Group. Conceptualized by the Board Chair and RHSC Director, this initiative was time-driven and top- down incubated. It was then outsourced to a communications firm for further development. In the process, potentially useful member input and involvement ended up being bypassed. This provides a good example of opportunities missed when donor driven and under time pressure. However well-conceived and timely H2H was, the members could have been more involved. On the other hand, the Secretariat does not currently have available funding to support a Working 110 Evaluation of the Reproductive Health Supplies Coalition Group for the significant demands on time and expertise that involvement with H2H would have necessitated. Secretariat as well as Executive Committee leadership would be enhanced by adding technical level strength. Members have suggested need for policy level expertise within the Secretariat, for example. A subject recently discussed within the SSWG was possibility of expanding the CARhs approach that alerts for stock-outs in countries-at-risk to a broader monitoring of commodity forecasts. Also suggested within this context was adding technical expertise to the Secretariat to coordinate such an effort. Similarly, several key informants suggested including different voices on the Executive Committee, including technical experts independent from affiliation with member organizations. 6. Considering Questions 7 and 9 in the previous section, what other changes in the Secretariat should be considered to improve the value-for-money of the RHSC? There is widespread recognition among members as well as within the Secretariat itself that that some higher levels of expertise at Secretariat level would be useful additions. At the same time, members hold the current staff in high esteem for all they are able to do to support RHSC operations and structures, such as Working Groups and the regional forums. The evaluators do not envision wide-scale expansion of current staff. Rather, judicious addition of a few positions, largely depending on future strategic directions, would be extremely useful. As the current Chair emphasized, form should follow function. For example, greater emphasis upon Global South participation and country-level activities would indicate adding country-level expertise to the Secretariat, perhaps even at the level of a deputy director. To increase capacity within the Secretariat, the donors would need to consider increasing the amount of core support currently provided to the RHSC. The $2.0 million annually available in donor funds is allocated across different expenses: salaries, office expenses, travel, convening meetings and conferences. Discussions with PATH as the fiscal agent and analysis of documents provided to the evaluators indicate that current funds are used efficiently. Given that this arrangement with PATH seems to be working well, now does not seem like the time for a change. Nor does it appear that there would be distinct advantages to registering and establishing the RHSC as an independent non-profit entity. Similarly, though some members saw advantages in locating the Secretariat elsewhere, such as London or within the Global South itself, again this does not seem the time for such a change. In the long run, if the RHSC is to become a permanent organization, it may be useful to consider such changes in status and location. For now, though, the evaluators see the RHSC as playing a particularly timely role for focusing global involvement with FP/RH upon subjects and countries of particular interest. It may or may not be in the long term viable to have the RHSC become a permanent standing organization. The funding for global programs could change considerably in coming years. Perhaps evolution towards regional or in-country approaches would serve future needs better. Success of both regional forums in raising funds during the last year may indicate different possibilities for the future. Furthermore, sectoral support within bi-lateral programs may call for different ways of organizing technical and financial assistance in the future. Ten years from now donor support for FP/RH and other development could have changed considerably. While the RHSC is needed now and for the next stage of FP/RH advancement, needs may well be quite different a decade from now. The following are offered as recommendations for strategically moving forward: 111 Evaluation of the Reproductive Health Supplies Coalition • Retain core focus on RH commodities There is clearly some pressure and interest within the Coalition to widen the focus from RH beyond its central focus on commodity security. The evaluation team recommends retaining the current focus on FP/RH. Although inroads have been made into repositioning the supplies agenda, there is still a long way to go and the work of the Coalition will be crucial to its achievement. However, the solution to encourage coordinated action to secure a slightly wider range of related maternal supplies is politically practical and practically achievable. • Maintain global as well as local focus on supply chain The question of whether the Coalition should operate more directly at country level was much discussed in the course of this evaluation. Various options were discussed for RHSC engagement further along the supply chain such as setting up mini-coalitions some countries and/or taking concerted action in a small number of selected focus countries. However there was no consensus, just a variety of opinions about how or what to do. The evaluation team therefore recommends that the RHSC maintain its focus as an organisation that primarily operates at global level, continues to reach toward countries through existing regional forums, and considers concentrated engagement of members in a select number of countries where that would add value. • Promote greater country level ownership of the Coalition More systematic approaches should be taken to increase country-level ownership of the work of the Coalition, making the RHSC more relevant to meeting country needs. At the moment the voices of southern partners are more or less absent from the Coalition in any sort of sustained way. With the exception of one co-chair, southern engagement with the Working Groups is virtually non-existent which means they miss out on much of the agenda setting. The Strategic Plan states: “While international expertise can play a critical role in recognizing weaknesses and identifying technical remedies, the search for long-term solutions will only succeed when countries themselves define the roles of their partners, including the Coalition, and set their own supply agenda. The Coalition will, therefore, reinforce the principles of country leadership and ownership in all country-level RH supply security activities”. 7. To what extent are the findings and recommendations of the 2008 sustainability study still relevant? How has the RHSC progressed, however incrementally, towards institutional and financial sustainability, and how can it further its progress over the next 3-5 years? • Considering the answers to Question 9 under “Structures/Processes,” what additional steps might the RHSC feasibly take to further strengthen its human and institutional capacities? • How can the RHSC improve the predictability of its financing? What are the pros and cons to further diversifying the RHSC’s funding, in particular beyond donor funding? The 2008 Sustainability Study was an early, useful exercise to externally appraise RHSC progress and prospects for financial sustainability. Undertaken a few years after formalization of the RHSC, the study occurred as the 2007-2015 Strategic Plan was being put in place and membership criteria were being expanded. The methodology applied to study sustainability of the RHSC was twofold: a review of selected Global Health Partnerships (GHPs) and interviews with Coalition stakeholders. In all, the study reviewed three GHPs (GAVI; PMNCH; and StopTB) and included interviews with 25 individuals from 16 RHSC member organizations. The purpose of the study was to assess approaches to sustain financial support for the RHSC Secretariat and Coalition operations. Given the connection of Coalition financing to the purpose 112 Evaluation of the Reproductive Health Supplies Coalition as well as activities of the RHSC, “added value” became a central theme of this study. Of particular interest was what activities of the RHSC are “over and above” what the members “would otherwise do”? The study concluded with a number of recommendations that have subsequently and successfully come to fruition. These include: collaborate with other GHPs; attract core support for the Coalition from joint funding by main stakeholders; do not pursue membership fees as a pathway toward RHSC financial sustainability; de-link financing of the Reproductive Health Interchange from that of the Secretariat; and include within the Secretariat’s budget resources for an Innovation Fund. Other major recommendations of the study which remain remarkably relevant to this evaluation are: • “Sharpen the focus of the Coalition…and ensure that the Strategic Plan reflects the added value of the Coalition” • “Include developing country government representatives systematically in Coalition activities” • “Keep the Coalition Secretariat small and efficient” The 2007-2015 Strategy remains intact as does the 2009 Terms of Reference which described the structure of the RHSC and usefully codified guiding principles for its operation. However, advice to sharpen and clarify the strategy, based on members input during the Sustainability Study, has not to date been followed. At present, as many key informants expressed during this evaluation, there remain disconnects and some confusion between the strategy as originally formulated and the present day activities and aspirations of the Coalition. Coalition membership expanded considerably since 2008, including increases in developing country representation. From a base of some 15 core member organizations in 2005, broadened criteria have resulted in a current membership nearing 200 organizations, with more joining every month. Though there has been some increase in developing country representation, including government as well as civil society members, these remain a small minority and the least active of Coalition members. While the Secretariat continues to be efficient with a small staff, more technical depth and some additional positions may be needed. 113 Evaluation of the Reproductive Health Supplies Coalition Annex 5 RHSC Member Survey 2012 RHSC Member Survey 2012 A report of results from RHSC members through Survey Monkey administered between April 24-30, 2012 Prepared by: Huong Q. Nguyen May 2012 114 Evaluation of the Reproductive Health Supplies Coalition TABLE OF CONTENTS Page Number Introduction 2 Survey Results 2-17 Background of Respondents 2 Participation in RHSC 2-3 Benefits of RHSC Membership 3-7 Country Level Engagement 7-10 What Does the Coalition Do Well 10-12 What Could the Coalition Do Better 12-17 Survey Limitations 17 Appendix 18-29 115 Evaluation of the Reproductive Health Supplies Coalition Reproductive Health Supplies Coalition (RHSC) Coalition Member Survey 2012 Results Report INTRODUCTION As part of the overall Reproductive Health Supplies Coalition (RHSC) evaluation, a member survey was developed and administered via Survey Monkey (an online survey platform) to Coalition members to inform the full evaluation. The survey was developed in collaboration with the two lead evaluation consultants and a sub-contractor. The survey consisted of 13 questions (with one question to opt out) and sent to a total of 200 respondents from the RHSC member list. Two of the respondents had already opted-out of any type of Survey Monkey questionnaires, leaving a total of 198 respondents. The survey was opened from April 24 to April 30, and reminder emails were sent out to alert members of the due date to fill in the survey. Of this number, 65 (out of 198) responded to the surveys, which was a 33% response rate. Responses were kept confidential to members of the Coalition. SURVEY RESULTS Background of Respondents The survey provided a snapshot of 65 Coalition members’ response to a six-day open survey administered online (via Survey Monkey). International NGOs were the highest number of respondents. Most respondents were mainly headquartered in high-income countries and focused their work in Anglophone Africa followed by Latin America and the Caribbean (LAC). Forty-percent (26 out of 65) of the respondents were from the Global South.10 The majority of respondents represented the family planning (83%) and reproductive health (69%) sectors. Many respondents also equally represented both HIV/AIDS (63%) and maternal and child health sectors (63%). Participation in RHSC The majority of respondents joined RHSC fairly recently, in 2010 (23%), and in 2006 (20%), which was when the Coalition was opened for membership. Many of the global south respondents joined between 2010 and 2011, ranging from in-country NGOs to government ministries to pharmaceutical manufacturers. Respondents joined RHSC for many reasons, but the top four included, 1) networking and sharing experiences with other groups (81.5%); 2) furthering an interest in FP and RH advocacy (70.8%); 3) learning about and contributing to commodity security (67.7%); and 4) making connections between FP, RH and MH (60%). Among the global south respondents, networking and sharing experiences as well as furthering an interest in FP and RH advocacy were also reflected as the top choices for joining the Coalition. However, many of respondents were not actively part of a working group or regional forum, as 37% said they do not sit in any of the working groups or regional forums. For respondents who were active, most listed the resource mobilization and advocacy (RMA – 26%) working group, and equally followed by both the Latin American and Caribbean (LAC-23%) Forum and the market development approaches (MDA-23%). The majority of the global south respondents _________________________ 10 Global South respondents are defined as an in-country or where the organization’s headquarters or a government ministry is located in a low/middle income country. 116 Evaluation of the Reproductive Health Supplies Coalition were part of a Working Group or Regional Forum. Among respondents who were from the Global South, participation was most prominent in the following: 1) LAC forum, 2) RMA and 3) MDA working groups. The frequency of participation in the Coalition varied with over 25% stating “at least once a quarter” and “at least once a month” to close to 14% choosing “on a weekly basis”. Only 6% of respondents reported that they were “inactive.” Respondents from the Global South follow the same patterns in their frequency of participation with 30% participating “at least once a quarter”, 23% “at least once a month” and 15% “on a weekly basis.” Benefits of RHSC Membership Members who responded described many benefits as a part of the Coalition. Top responses included: information-sharing, technical assistance and learning; addressing supply/commodity issues; networking and staying connected; enhancing global advocacy for FP/RH and commodity security; leveraging funding and resources; and deepening partnerships. Many of these benefits are inter-related/inter-dependent. Quotes are provided to highlight the open- ended responses to highlight each category. One quote highlighted here describes the many inter-related benefits derived from membership: “My organization's participation in the Coalition's activities has opened us to a lot of insights into global level advocacy for family planning and reproductive health. Being involved in the RHSC has improved our ability to attract more funding to implement country level advocacy strategy. For instance, through the RMA-WG, my organization is currently a beneficiary of the Small Grant. It has also enhanced our visibility both at national and global levels, as far as advocacy is concerned. It has also strengthened our credibility and deepened our partnership with in-country key stakeholders in family planning and reproductive health.” (In-country NGO) Information-sharing, technical assistance, and learning. Respondents benefited from information-sharing, technical assistance and learning among members, as well as through participation within the working groups and regional forums. Exchange of information and learning from others’ experiences encompassed learning about FP/RH and commodity security issues, what others are doing across geographies, the latest updates and news, and new ideas and strategies for advocacy that can leverage and broaden the impact of members’ work. “Although we are far away from all meetings and activities, being a member places us in the fold of information and sharing.” (Pharmaceutical Manufacturer) “Participation in the Francophone forum has helped us understand what other groups and efforts are underway in Francophone Africa and opened up some opportunities for leveraging and learning (e.g., LAPTOP).” (INGO) “As a participant of the LAC Forum we get information of new actions in the field that helps us to be informed and to ensure we are doing the best practices.” (In- country NGO) “It has [helped] to get relevant information and an [in]sight into the nature of our work including providing with the needed resources for running some projects.” (In-country NGO) “We simply follow all your e-mailings and news to keep abreast of family planning, reproductive health and maternal health developments. Always looking 117 Evaluation of the Reproductive Health Supplies Coalition for opportunities in which a very small organization like our own in Venezuela can participate or benefit.” (In-country NGO) “Involving my organisation in others initiatives regarding to the family planning and reproductive health field. Sharing also our experience in this field with coalition members.” (Inter-Government Organization) “The RHSC has provided a platform for discussing contraceptive quality assurance and WHO prequalification and for developing an evolving strategy around these issues which are of great importance to my organization. The same is true for total market initiatives--the RHSC provides a platform for sharing information strategies, lessons learned etc. The opportunity for discussion around issues such as these has strengthened the work of my organization and also enabled us to share our learning with others. (INGO) “To be part of the working group has given us opportunities to explore the development of activities/programs with other organizations that are part of the Coalition or to have information about the development of new initiatives by the working groups that could be in favor of our member affiliates.” (INGO) “[We] became involved with the Coalition in the early days through our RH procurement work for DFID. We have been involved in the Systems Strengthening Working Group (including contributions to the RHInterchange, Professionalization of Supply Chain Managers and other initiatives) as well as providing technical inputs into the Market Development Approaches Working Group. The Coalition has helped us to share best practices in health supply chain management and engage with a variety of actors all driven by efforts to improve RH commodity security, including governments, social marketers, NGOs, the private sector and funders.” (Consulting Firm) Addressing Commodity/Supply Issues. Respondents stated that the Coalition convenes diverse partners and provides members with technical assistance and opportunities to interact and collaborate with other stakeholders in both the private and public sector to address supply issues. Planning and Procurement “Using the RH Interchange helps our contraceptive procurement planning.” (INGO) “Learning about the quality of medicines initiatives supported by the RHSC which has helped our organization plan drug introduction and procurement.” (In-country NGO) “Being involved in the Coalition helped us resolve a long pending issue related to a Sub-Saharan country, where generics were unable to make any headway despite all efforts. In the PPD meeting in India around 2005, we could meet the Minister of Health of the country along with UNFPA Procurement Head and actually resolve the issue.” (Pharmaceutical Manufacturer) 118 Evaluation of the Reproductive Health Supplies Coalition “It has helped in sharing challenges met in procuring some of key commodities for RH, including lack of quality assured products for some commodities such as contraceptives. It has contributed to work toward harmonized approach for QA of health products and how to move forward. It provided information and contacts on innovative funding mechanism such as the Pledge Guarantee for Health to prevent treatment disruption and enable better procurement outcomes. It helped in understanding how the Country at Risk (CAR) mechanism put in place to prevent stock out situation for contraceptives and better coordinate donor support for family planning.” (Multilateral Development Agency) “We are able to use the RH Interchange to make logistics decisions. The newsletter brings out also to lessons to learnt from which can be applied to our home situation.” (Government Ministry) Lowered Pricing “The work of the coalition to reduce the price for Implanon has helped to contain the cost of commodities. The RHSC Meeting in Addis last year provided for me an excellent networking opportunity. Partnership and long-lasting contacts originated from that meeting.” (INGO) “The price reductions announced by Merck in the Ethiopia meeting demonstrated the power of linking the private sector with the coalition--donors, procurers, and country leaders. The coalition held a Webinar on contraceptive stock outs and galvanized 80 members to provide feedback on a Gates Foundation assessment that increased the dialogue on this critical issue and launched new work in this area in Nigeria and Senegal.” (Private Foundation) “The RHSC has helped to negotiate lower product prices, which has enabled us to procure more commodities at lower prices. The RHSC has enabled closer coordination on contraceptive supply across countries and has helped to identify supply shortfalls (and oversupply) in a timely fashion and to remedy in many instances. The RHSC has helped to elevate contraceptive supply issues higher on international agenda. The RHSC began the initial discussions of innovative financing approaches that laid the groundwork for PGH and Tier 2 pricing partnerships.” (Bilateral Development Agency) Networking and staying connected. Respondents described the benefits derived from being networked and staying connected to others who are working on the same issues, sharing experiences, strengthening their work, and having the opportunity to meet and be in touch with members they usually do not interact with that can lead to opportunities for collaboration and partnerships. “The Annual meetings of RHSC provide a good opportunity to network and collaborate with networks and organisations in other parts of the world. Via face to face meetings, it strengthens networks and initiates collaboration. The RHSC staff have been invited to some of our meetings to provide technical insight in supply chains and opportunities and challenges linked to it. This has been very helpful for our partners.” (INGO) “RHSC has allowed us to meet people and organizations that we do not regularly meet or interact with.” (In-country NGO) 119 Evaluation of the Reproductive Health Supplies Coalition “It is exciting for a fairly new organization like ours to connect with a global network of people and groups who share our concern about expanding RH commodity access and security. We hope to become more active in the Coalition as we grow.” (INGO) “Networking with key opinion leaders in the field of Family planning.” (Pharmaceutical Manufacturer) “It has brought us in touch with other entities.” (INGO) “Mostly just staying connected and hearing what others are up to.” (Private Foundation) Enhancing advocacy efforts. Members elaborated on how being a member reinforced their organization’s commitment to FP and commodity security, leveraged the work of the Coalition to gain traction in advocacy, received insight into current and upcoming global advocacy and activities, as well as enhanced organizational visibility and credibility to engage in advocacy for FP/RH and commodity security. “Being a member of RMA working group strengthened our advocacy for family planning especially when project RMA was stilling running. A strong network of CSOs was formed under RMA and it was coordinated by my organisation a factor that made us achieve a great deal as far as advocacy for RH commodities is concerned. We managed to engage government and increased the budget of FP commodities by almost three folds.” (In-country NGO) “The networks has provided us with recent information and new angles in the areas of reproductive health and family planning and this we have been able to use to localise them for advocacy use and creating awareness among the people in Sub-Saharan Africa through our networks.” (Development Media Organization) “As one of the founding organizations of the Reproductive Health Supplies movement, we have worked via the structure of the RHSC to further all of our advocacy objectives -- for example by using the working group structure managed by the RHSC to build a community of advocates and country NGOs active on the issue. The small grants mechanism has enabled us to raise awareness of RH supplies and build champions through the production of an advocacy film.” (INGO) “It helps us get our message(s) out.and we meet with other organizations, private sector and donors. Our corporate group comes from the RHSC.and MHSC Wkg. Grp.” (INGO) “Relationships through the RMA WG have benefited our work in many areas. The Coalition has also provided a platform for supplies advocacy.” (INGO) “Advocacy on female condoms.” (INGO) 120 Evaluation of the Reproductive Health Supplies Coalition Building and Strengthening Partnerships and Relationships. Participation in the Coalition offers the possibility to develop new partnerships with different stakeholders and strengthen existing partnerships and collaborations. “We are learning more about contraceptive security through the Coalition; and beginning to find opportunities to link with others in the group to help our members strengthen their contraceptive security. Just beginning this but am very excited by the opportunities that are opening up to our organization.” (INGO) “Participation in the 2011 Conference resulted in meeting IPAS, an organisation that focuses on abortion services. The interaction has resulted in developing a strong partnership between YONECO and IPAS and currently implementing the SRH and R programmes for young people in the communities.” (In-country NGO) "I had the opportunity to be in a regional meeting and there I could know the big political and technical discussion about supplies. As part of this process, we have included in the activities plan of CLACAI an initiative to incorporate the decision makers, Deabogacia, in drugs and try to influence so that they can include in their agendas the supplies of birth control and misoprostol. As a result of this work, we CLACAI, organized a meeting in Mexico, with the specialists in drugs of 4 countries. This was in the end of the year 2011." (Regional networks of individuals and NGOs) “The Coalition has been a neutral place to host meetings, house tools, commission studies, etc., and a place where competitors can actually collaborate. It is useful to USAID, and to USAID CAs, to have such a place.” (Consulting Firm) Leveraging and Influencing Funding Streams and Resources. Members stated opportunities to attract and leverage targeted and increased funding for commodity supplies. “First and most importantly, is the exposure with other international members of the coalition through the meetings, shared materials and the newsletter which we do share frequently. The opportunity to compete for funding though we have never got yet, but is another chance to learn from our shortcomings, as we look for funds to implement our activities.” (In-country NGO) “Helped deliver on commitments to reduce the price of RH supplies.” (Bilateral Development Agency) “Participation to the RHSC has helped to streamline our funding to SRHR and Commodities.” (Bilateral Development Agency) “Fund small grants program with RMA Working Group and support variety of other activities, most of which are tied to bringing a developing country perspective.” (University) “Educating and informing RHSC members about SC issues. Helping to improve the responsiveness of donors to country needs for supplies through the CARhs.” (Pharmaceutical Manufacturer) 121 Evaluation of the Reproductive Health Supplies Coalition Country Level Engagement Top Country Engagement Activities. The majority of respondents (54 total) agreed that the Coalition should engage more at country level to expand FP and commodity security. The top three activities (some respondents chose more than the three choices requested) prioritized for country level expansion include: 1) Resource mobilization and financing for RH commodities (55.6%) 2) Advocacy to increase demand for and access to FP/RH services and supplies (50%) and Supply chain strengthening for RH commodities (50%) 3) Advocacy to ensure RH security (48.1%) Among the Global South respondents, the prioritizations were somewhat different: 1) Advocacy to increase demand for and access to FP/RH services and supplies 2) Resource mobilization and financing for RH commodities 3) Private sector growth for RH commodities and assistance to similar in-country groups/networks to promote FP/RH coordination Other Global South responses to expand to country level activities included: “There is a need for such coalition to work at country level. This will help nearly all the countries to have a professional approach to the issue of family planning. Which at times hangs between high sounding philosophy and simple rhetoric.” (Pharmaceutical Manufacturer) “Assistance to provide trainings and funds to enable us to move. We need to expand in rural areas and provide us the resources center.” (Community based organization) A few respondents noted to leverage the strength of the Coalition and that moving towards country level engagement should be strategic and intentional in order to enhance impact: “The early success of the RHSC has been focused on improved coordination, raising commitment, and the development of innovative supply approaches at the HQ level. There is an opportunity to amplify the impact of the RHSC by strengthening coordination, commitment, and innovation at the country level. This is happening already in some countries, but the RHSC could make these efforts more intentional and expand effective approaches to more countries.” (Bilateral Development Agency) “I am fully supportive of the Coalition being active in 2-4 countries and testing whether they can build a comparative advantage that serves the field.” (Private Foundation) “It actually depends. If the Coalition is able to get the ears and eyes of the decision makers in key countries where the CPR is lower than, say 25%, then possibly an in-depth engagement at country level may not be required. Having said that I feel the focus of the Coalition should be on the countries mentioned above - where the CPR is lower than 25%. Focus and making it successful will enhance credibility and enable to focus on the second set of countries - with CPR between 25 and 50%.” (Pharmaceutical Manufacturer) 122 Evaluation of the Reproductive Health Supplies Coalition Some of the “other” responses varied. A few respondents felt that the Coalition’s working groups, country member organizations/affiliates and the regional forums already engage at the country level, and are more appropriate mechanisms to engage in consultation and coordination at the country level. One respondent stated that the Coalition needs to be better defined – is it the Secretariat or the RHSC members? - in order to determine at what level to engage in. “From my perspective, the question is not "if" but rather how. Already the Francophone and LAC forums are focusing more specifically on country needs and providing a platform for the development of country specific initiatives. Activities undertaken through the Coalition are largely undertaken by members and member organizations must be responsive to the requirements of their funding (whether funding comes from government budgets or donors). One mechanism would be for donors who are members of the Coalition to jointly underwrite specific and coordinated activities at country level implemented by member organizations. This has been done at the global level in the cases of the PGH and AccessRH, but not at the country level. Such an approach would imply a different approach to funding. Additionally, it would be useful to work with existing country members on a more in depth level to find out what they expect from RHSC membership and how they see it strengthening RHCS in their countries.” (INGO) “The question is unclear in that it does not adequately identify what is meant by the term 'Coalition' - is it the RHSC Secretariat or RHSC members. RHSC members are already active at national level.” (Other) Why RHSC SHOULD NOT or CAUTIONED NOT to expand to country level engagement However, a handful of respondents, mostly from INGOs, cautioned not to move towards country level engagement due to the existing comparative advantage of Coalition country members, working groups and regional forums that are already undertaking country work, as well as the time and resources it would require to undertake this effort. The respondents felt the Coalition should focus on its strength as a global and regional coordinating and convening mechanism instead. Others questioned the need to better define “country level engagement”, the RHSC structure and membership’s roles/responsibilities, and if membership expansion to more “southern” representatives is needed to engage in this. • Responses were varied, but the top reasons that surfaced were the following: o RHSC should focus on the big picture, at the global level, and let the RHSC members (organizations and affiliates), working groups or regional forums take initiative or take the lead to coordinate efforts at country level. There are already informal groups and RHSC members who are doing advocacy at the country level and know the context better. o RHSC should concentrate its efforts on coordination, global and regional advocacy, engage new partners, and support innovation at the macro-level. o RHSC needs to look at long-term funding commitment and sustainability before engaging in country level efforts. Is the funding going to be there to support in- country activities? Would RHSC be overstretched, given its lean staffing structure, time and resources it would require? o RHSC should consult with country members on how best to do this. • Other questions came up: 123 Evaluation of the Reproductive Health Supplies Coalition o How does RHSC define “country-level engagement”? Does it mean expanding membership to southern countries? Does it mean taking a specific activity in a specific country? How is the Coalition uniquely positioned for this activity and how it will achieve it? Are the regional fora the best way to engage at country level? o What is the value-added that RHSC can bring to national efforts? Some quotes highlight these outcomes: “No, the RHSC should not engage more at the country level. RHSC members are engaged at the country level and the RHSC should allow its member organizations to take the lead in country level initiatives. The RHSC should play to its strengths around coordination, global and regional advocacy and engaging new partners.” (Pharmaceutical Manufacturer) “I do not feel the Coalition should compete with its members. It's a delicate dance… The RHSC shouldn't do those things - - RHSC member organizations should do them. As a membership coalition of technical, advocacy, service delivery, government, and funding organizations, the RHSC should not compete with the work of its members, it should facilitate and "further" that work. The RHSC should be a place where members come together and set a common agenda and hold each other accountable for progress.” (Consulting Firm) “…the Coalition in my opinion should work much more at a macro level. Important to keep in mind that the work at the country level by the working groups of the forums will require economical resources in order to be able to develop concrete activities that contribute to the implementation of policies and changes in policies that would benefit the population with less income.” (INGO) “In my opinion, the topics listed apply much more to the working groups of the Forums, although I think that the Coalition should include into their plans to work in the LAC Region and not to concentrate in Africa and Asia only. It is also important to include organizations of the civil society, such as NGOs and not to direct all the efforts and resources to governments…Base on the above comment, the Coalition in my opinion should work much more at a macro level. Important to keep in mind that the work at the country level by the working groups of the forums will require economical resources in order to be able to develop concrete activities that contribute to the implementation of policies and changes in policies that would benefit the population with less income.” (INGO) What does the Coalition do well? Playing a global convening and coordinating role (providing a common platform) that brings different stakeholders and constituencies together and builds consensus to mobilize for resources and bring visibility to RH commodity security. Good leadership from John Skibiak was mentioned as holding this together. The strength of this allows the following activities to be undertaken to achieve the Coalition’s goals and strategies: • Offering a platform for networking among a wide range of international stakeholders, including existing members, key players and bringing in new partners (linking people together, esp. private and public sector, country governments, donor engagement, engaging manufacturers to look at “social pricing”; supporting dialogue between N-S; bringing in southern representation, voices and perspectives), which 124 Evaluation of the Reproductive Health Supplies Coalition provides a broad range of perspectives, particularly from the “Global South,” to address supply issues. • Providing joint advocacy and mobilization efforts (collecting and coordinating diverse views into a “representational set”- building consensus; changing/influencing the public debate about FP and RH; being strategic in its interventions; giving information about advocacy; leveraging donors; creating demand to counter supply side of the global market for contraceptives) that elevate the visibility of commodity security and its role in development. • Information-sharing, technical assistance and learning (knowledge management) - facilitating the exchange of information and experiences; documenting problems with RH commodity access and security; monitoring current status of commodities at country level (RH Interchange); keeping track of policy issues and funding opportunities; thinking out-of-the-box (innovation); providing technical assistance via workshops to enable members to address RH commodity issues. • Providing consistent communication with RHSC members and partners of current and upcoming Coalition activities and opportunities to engage in RH supplies issues/advocacy through the email listserv, newsletters (Supply Insider), updates on its website, annual meetings, and working groups are well-organized. Quotes below describe the range of impact of the Coalition: “The coalition has established an effective network and platform among all partners.” (Government Ministry) “The coalition is very good at information sharing and involving all players at all levels. It is also very good at organizing international meetings that a lot of information is shared that helps to advance advocacy for RH commodities at country level.” (In-country NGO) “The Coalition offers perspective and support for the vast need and forward public health movement in family planning, reproductive health and maternal health.” (In-country NGO) “The constant update of what is taking place to the membership. Engagement with other stakeholders in advancing the cause for reproductive health supplies especially for the southern hemisphere.” (In-country NGO) “Keep track of all policy issues and new funding opportunities.” (In-country NGO) “The RHSC provides a critical central convening and coordination role. Every organization that is involved in reproductive health supplies/contraceptive security participates in or work via the RHSC. The RHSC does not have parallel structures or competing groups or coalitions -- and that is unique. This puts the RHSC in an enviable position -- and one where they have considerable impact on the direction of the movement. What the RHSC does well is to rely on all of its member to implement and magnify the work -- something the RHSC does not have the resources to accomplish itself.” (INGO) “Create a space for the various actors concerned with contraceptive supplies to work together in ways that multiply their effectiveness and impact. As such, it has 125 Evaluation of the Reproductive Health Supplies Coalition engendered "out of the box" thinking on innovative approaches to maximize impact.” (Bilateral Development Agency) “Brings together a wide range of stakeholders involved in FP / RH work, including governments, NGOs and the private sector as well as a mix of implementers, researchers and policymakers. This enables cross-fertilization and lesson learning, which in the end strengthens all those involved.” (Consulting Firm) “The Coalition has leveraged the debate on family planning and reproductive health and is also helping the partners to intervene in a very strategic manner. In my view the Coalition has done well.” (Development Media Organization) “Makes the issue of supplies visible to the global community and, through its working groups, helps elevate the voices of in-country advocates.” (INGO) “Communicate with members about current and upcoming Coalition activities and opportunities to be engaged in RH supplies issues/advocacy. Leveraging of donors and partners at a global level to ensure adequate funding for commodities as well as responding to specific country needs”. (RHInterchange, CARHs). “Monitoring current status of commodities at country level. Listening to different perspectives and having an openness to new ideas and ways of working.” (INGO) “Partnership and share with coalition members some of the initiatives going on. Building consensus among coalition members on the initiatives regarding to the reproductive health and family planning issues.” (Inter-government Organization) “Provide technical expertise on supply chain management, which is not always known to all SRHR advocates; early warning system for stock outs in country.” (INGO) What could the Coalition do better? Funding and Staffing. Raise more funding to support the need to expand staffing of RHSC who can help coordinate, implement and expand the Coalition’s activities, especially to support the working groups and to coordinate in-country level activities. Advocacy and Messaging. Improved communication/messaging for advocacy. Be more specific by engaging in more targeted advocacy, specifically targeting governments, donors and private sector. • Have clarity in the purpose as well as messaging (RH and supplies are important). Have clear communication so to not let linkages to MCH or HIV diffuse our messages. • Have tighter, crisper messaging. • Develop more advocacy messages that comes out regularly to keep the issue visible in the public domain. Membership Representation • Finding more ways to increase members’ participation and engagement in the Coalition. This may mean being more clear (better defining) about expectations of membership as 126 Evaluation of the Reproductive Health Supplies Coalition well as clarity around specific needs for collective action. Get members more active in working groups. • Increase global south representation by engaging with southern governments, local NGOs, NGOs that need more advocacy support, etc. and providing them with more resources (grants) and technical assistance. Involve more country level representation in working groups. • Engage new partners and allies. Support new partnerships. Take more proactive role in bringing in partners to drive progress on RH security. o Bring in more private sector and technical experts to help facilitate the work. o Build stronger ties with emerging economies as they will influence the aid architecture (e.g. China for generic manufacturing). Member Support • Membership Support o Provide more capacity-building to members, especially smaller NGOs and southern countries, on advocacy and commodity management. o Set up resource centers, trainings and workshops. o Assist more members with financial support to participate and attend more meetings. Pay more attention to country needs by supporting those coalition members’ work. o Increase organizational grant size relative to proposed activities and have more funding activities. • Member Communications o Have more organized and systematic (predictable) communications with members – sending out emails, reminders for meetings, etc. on a regular basis (weekly, monthly, etc.). Provide more updates on activities and progress. Update the Coalition’s contact list. o Share more information, especially on updated tools, resources, publications, etc. o Be more transparent about the “inner workings” of the Coalition. o Take members concerns into consideration o Strengthened grounding of what country members think RHSC can do to address RH security issues in country o Linking membership feedback to key international stakeholders - Loop membership feedback back to UNFPA Working Groups Support • Better coordination of working groups (WGs) to achieve broader impact o Make sure WGs mutually understand and support each other to benefit the whole o Provide better links between the WGs o Better at supporting win-win synergies among Coalition members and WGs o Draw on members’ comparative advantage vis-à-vis the issue for joint focused efforts Advocacy Strategies • Expansion to regional efforts and in-country work o Support South-South partnerships and advocacy o Regional efforts 127 Evaluation of the Reproductive Health Supplies Coalition o Not explored regional efforts, especially focusing on MICs (middle income countries) that are experiencing or experienced donor phasing out but still have low CPR o Have regional meetings o Include LAC region in the plans o In-country o Focus on 5-7 countries with low CPR (under 25%) o Work in a few countries to test and scale-up o Support country-based groups to develop localized action plans on various topics o Local engagement on supplies - engage country members to build capacity in commodity management o Provide concrete TA and financial assistance for country needs o Guide countries in implementing policies in a more pragmatic manner • Support for new FP methods/technologies and quality of medicines/supplies o Make the case for new (esp. non-hormonal) FP methods that are acceptable in communities o Explore manufacturing FP methods that are not affordable o Focus on quality of medicines/supplies • Linking RH commodity issues and learning from other health issues/sectors o Linking RH to maternal health issues o Leveraging RH to wider health issues o Look at other health communities and adapt lessons from commodity security practices (GAVI model – any synergies with GAVI and advantages for advancing SRH?) Operational Focus o RHSC is spread out thin. Should focus on a few specific issues o RHSC comparative advantage is the technical expertise on supplies issues and should focus on it. Perhaps it should not focus too much on general advocacy for FP and SRHR. Below are quotes that highlight respondents’ broad range of suggestions for improvement: Operational: Additional Funding and Staffing “Coalition cannot do too much more without additional funds and staff. If they had more resources the number one priority would be to get stronger commitment of donors for contraceptive security, i.e., more active advocacy at the global level.” (University) “I feel that the RHSC secretariat needs to expand its staff (while staying lean) to enable the various existing work streams to move forward more expeditiously, to take on some of the new tasks that have evolved recently (including commodity pricing, hand-to-hand related advocacy/monitoring), and to enhance the RHSC's ability to support activities at the country-level.” (Bilateral Development Agency) “Feature the importance of advocacy for supplies. Become less process oriented and more outcome oriented.” (INGO) Structures and Processes: Working Groups and Regional Forums 128 Evaluation of the Reproductive Health Supplies Coalition “The Coalition could be much better at making sure that the Working Groups mutually understand and support each other’s work for the benefit of the whole. The Secretariat could better understand and support advocacy. The Secretariat and Executive Committee could be much better at supporting win-win synergies among Coalition members.” (INGO) “The link between the different working group streams could be done better, e.g. RMA WG advocating for issues MDA or SSWG are working on. However, since all is on a voluntary basis, it is difficult to free time of advocates, already burdened with lots of other work. The challenge is to advocate for RH commodities specifically, since it is very specific, whereas most advocates and donors are working on a higher more general level. Maybe we should think who should be the target group for advocacy? Maybe it would make more sense to target governments and private sector in country and donors in de delegations who are actually working on supply chain management on a daily basis. It is important that there is overall advocacy, but there are enough organisations advocating for SRHR or FP or RH. The true strength of the RHSC lies in its technical expertise on supplies, so maybe it should think more about how to get that expertise across rather than being involved in the more general issues of overall SRHR advocacy.” (INGO) “To include the LAC region in their plans.” (INGO) Membership and Partnership “Ensuring all members are at least member of any one of the working groups - Find ways of making members interact on a more regular basis.” (Government Ministry) “Get more members to engage. but this is a common problem that many face. perhaps when organizations are applying for membership, a greater emphasis on active involvement should be made.” (INGO) “Reach out to in-country groups that need support for FP/RH/CS advocacy.” (INGO) “I think that the in-country coordination with Civil society will increase and find better ways to share responsibility and make work more effective.” (Government Ministry) “The Coalition could increase the input on capacity building. The Coalition should focus on generating more opportunities on capacity building for members.” (Government Ministry) “Improve the dissemination of technical information and promote greater exchange/collaboration. Also, it should broaden its horizons to other components, especially those linked to safe abortion.” (Regional networks of NGOs and individuals) “Recognize that the architecture of international aid will change and build stronger links with the emerging economies (especially China) that are the likely primary source of the generic contraceptives we need.” (University) “Coalition depends on volunteers to do the work of the working groups and we could get so much more done if we had full time top notch experts working with each group. Bringing in the 129 Evaluation of the Reproductive Health Supplies Coalition private sector financing to link organizations with PRIs and other guarantees would be ideal. Being more active in a few countries to do both advocacy and contraceptive stock out activities (testing and scaling) is critically needed.” (Private Foundation) “Work with other similar health communities (HIV/AIDS, Malaria, TB, Nutrition, Maternal Health) to share and adapt lessons from commodity security work in RH, which is more mature and well developed than other health work. Support for country programs should look at leveraging RH activities for wider health, rather than remaining ‘vertical’.” (Consulting Firm) “Engage new allies and partnerships, not only in the corporate space. Support on-going participation of in-country partners.” (INGO) Communications and Messaging “I think the RHSC could improve, tighten and expand its messaging. In addition, the RHSC needs to put more resources and effort into the development and involvement of countries -- mobilizing resources and building the capacity of country structures to take on the outreach, advocacy and implementation work. This is not an easy task, but is critical for long-term sustainability given the movement of large funders towards investing in countries themselves and the dwindling of support for international NGOs based in developed countries.” (INGO) “Organize the way information is shared with all coalition members. It´d be better if information is distributed on a regular basis (monthly, weekly, etc.) and not randomly as it is now.” (In-country NGO) Perspectives from the Global South “I think the coalition should improve on their support to small local NGOs especially as it relates to capacity building in various areas.” (In-country NGO) “Paying more attention to the countries needs supporting them through coalition members meeting those needs.” (Inter-Government Organization) “Assist more members who do not the financial to be part and participate in important coalition meetings.” (In-country NGO) “Coalition can actively involve country level local organization in most committee level to strengthen their capacities.” (In-country NGO) “Full engagement of the southern membership partners despite their level in form of size, and influence.” (In-country NGO) “The coalition should mobilize more resources for grants to South to advance advocacy for country ownership of commodity security.” (In-country NGO) “To set up resources centers, trainings, and funding.” (Community based Organization) “Local engagement on supplies.” (In-country NGO) 130 Evaluation of the Reproductive Health Supplies Coalition “Develop country based groups to discuss and present the localized action plans on various topics.” (Pharmaceutical Manufacturer) SURVEY LIMITATIONS The survey received a 33% response rate (65 responses out of 198 total respondents) from a 7- day open online survey platform. Thus, the results of this survey may not be reflective of the larger membership, and are only reflective of one-third of the Coalition’s membership. Nevertheless, the information provides a rich snapshot of members’ views and assessment of the Coalition. The responses are used to inform the full evaluation of the RHSC. (Note: Quotes are taken directly from members’ open-ended responses.) 131 Evaluation of the Reproductive Health Supplies Coalition APPENDIX Q1. Which of the following categories describes your organization? Answer Options Response Percent Response Count Bilateral development agency 4.6% 3 Multilateral development agency (including health partnerships) 1.5% 1 Government ministry 4.6% 3 International NGO 33.8% 22 In-country NGO 18.5% 12 Pharmaceutical manufacturer 10.8% 7 Institutions of higher learning (University or other academic institution) 6.2% 4 Private Foundation 3.1% 2 Research/think-tank 1.5% 1 Consulting firm 4.6% 3 Other (please specify) 10.8% 7 answered question 65 skipped question 0 Q2. Where does your organization primarily focus its work? (please specify all that apply) Answer Options Response Percent Response Count Africa – Anglophone 70.8% 46 Africa – Francophone 41.5% 27 Asia 49.2% 32 Bilateral development … Multilateral … Government ministry International NGO In-country NGO Pharmaceutical … Institutions of higher … Private Foundation Research/think-tank Consulting firm Other (please specify) 0.0% 10.0%20.0%30.0%40.0% Which of the following categories describes your organization? Which of the following categories describes your organization? Which of the following categories describes your organization? Which of the following categories describes your organization? Bilateral development agency Multilateral development agency (including health partnerships) Government ministry International NGO In-country NGO Pharmaceutical manufacturer Institutions of higher learning (University or other academic institution) 132 Evaluation of the Reproductive Health Supplies Coalition Europe 16.9% 11 Latin America and Caribbean Region 52.3% 34 North Africa/Middle-East 26.2% 17 Other (please specify) 21.5% 14 answered question 65 skipped question 0 Q3. Is your organization headquartered in a low/middle or high income country (irrespective of where you, personally, work)? Answer Options Response Percent Response Count Low/middle income 38.5% 25 High income 61.5% 40 answered question 65 skipped question 0 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% A fr ic a – A n g lo p h o n e A fr ic a – F ra n c o p h o n e A s ia E u ro p e L a ti n A m e ri c a a n d C a ri b b e a n R e g io n N o rt h A fr ic a /M id d le - E a s t O th e r (p le a s e s p e c if y ) Where does your organization primarily focus its work? (please specify all Where does your organization primarily focus its work? (please specify all Where does your organization primarily focus its work? (please specify all Where does your organization primarily focus its work? (please specify all that apply)that apply)that apply)that apply) 133 Evaluation of the Reproductive Health Supplies Coalition Q4, When did your organization become an official member of the Reproductive Health Supplies Coalition (the Coalition)? Answer Options Response Percent Response Count 2006 20.0% 13 2007 12.3% 8 2008 10.8% 7 2009 12.3% 8 2010 23.1% 15 2011 13.8% 9 2012 7.7% 5 answered question 65 skipped question 0 38.5% 61.5% Is your organization headquartered in a low/middle or high income Is your organization headquartered in a low/middle or high income Is your organization headquartered in a low/middle or high income Is your organization headquartered in a low/middle or high income country (irrespective of where you, personally, work)?country (irrespective of where you, personally, work)?country (irrespective of where you, personally, work)?country (irrespective of where you, personally, work)? Low/middle income High income 134 Evaluation of the Reproductive Health Supplies Coalition Q5. What sector does your organization work in? (please specify all that apply) Answer Options Response Percent Response Count Family planning 83.1% 54 Reproductive health (includes safe abortion access) 69.2% 45 HIV/AIDS 63.1% 41 Maternal and child health 63.1% 41 Other (please specify) 27.7% 18 answered question 65 skipped question 0 2006 2007 2008 2009 2010 2011 2012 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% When did your organization become an official member of the Reproductive When did your organization become an official member of the Reproductive When did your organization become an official member of the Reproductive When did your organization become an official member of the Reproductive Health Supplies Coalition (the Coalition)?Health Supplies Coalition (the Coalition)?Health Supplies Coalition (the Coalition)?Health Supplies Coalition (the Coalition)? 2006 2007 2008 2009 2010 2011 2012 135 Evaluation of the Reproductive Health Supplies Coalition Q6. Which of the following best describes why your organization participates in the Coalition: (please specify all that apply) Answer Options Response Percent Response Count Learning about and contributing to commodity security 67.7% 44 Furthering an interest in family planning and reproductive health advocacy 70.8% 46 Ensuring that family planning and reproductive health products are linked with service provision 50.8% 33 Making connections between family planning, reproductive health and maternal health 60.0% 39 Networking and sharing experiences with other groups 81.5% 53 Leveraging additional funding and resources for your work 43.1% 28 Other (please specify) 12.3% 8 answered question 65 skipped question 0 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% Family planning Reproductive health (includes safe abortion access) HIV/AIDS Maternal and child health Other (please specify) What sector does your organization work in? (please specify all that apply)What sector does your organization work in? (please specify all that apply)What sector does your organization work in? (please specify all that apply)What sector does your organization work in? (please specify all that apply) 136 Evaluation of the Reproductive Health Supplies Coalition Q7. Does someone from your organization sit on any of the following: (please specify all that apply) Answer Options Response Percent Response Count Coalition’s Executive Committee 12.3% 8 Systems Strengthening Working Group (SSWG) 18.5% 12 Market Development Approaches (MDA) Working Group 23.1% 15 Resource Mobilization and Advocacy (RMA) Working Group 26.2% 17 Caucus on New and Underused RH Technologies 15.4% 10 LAC forum 23.1% 15 Francophone forum 12.3% 8 None of the above 36.9% 24 answered question 65 skipped question 0 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% L e a rn in g a b o u t a n d c o n tr ib u ti n g t o c o m m o d it y s e c u ri ty F u rt h e ri n g a n in te re s t in fa m ily p la n n in g a n d re p ro d u c ti v e … E n s u ri n g t h a t fa m ily p la n n in g a n d re p ro d u c ti v e h e a lt h … M a k in g c o n n e c ti o n s b e tw e e n f a m ily p la n n in g , re p ro d u c ti v e h e a lt h … N e tw o rk in g a n d s h a ri n g e x p e ri e n c e s w it h o th e r g ro u p s L e v e ra g in g a d d it io n a l fu n d in g a n d re s o u rc e s f o r y o u r w o rk O th e r (p le a s e s p e c if y ) Which of the following best describes why your organization participates in Which of the following best describes why your organization participates in Which of the following best describes why your organization participates in Which of the following best describes why your organization participates in the Coalition: (please specify all that apply)the Coalition: (please specify all that apply)the Coalition: (please specify all that apply)the Coalition: (please specify all that apply) 137 Evaluation of the Reproductive Health Supplies Coalition Q8. How would you describe your frequency of participation/activeness in the Coalition? Answer Options Response Percent Response Count On a weekly basis 13.8% 9 More than once a month 9.2% 6 At least once a month 24.6% 16 At least once a quarter 27.7% 18 More than once in a year 7.7% 5 Once per year 7.7% 5 Less than once per year 3.1% 2 Inactive 6.2% 4 answered question 65 skipped question 0 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% C o a lit io n ’s E x e c u ti v e C o m m it te e S y s te m s S tr e n g th e n in g W o rk in g G ro u p (S S W G ) M a rk e t D e v e lo p m e n t A p p ro a c h e s (M D A ) … R e s o u rc e M o b ili z a ti o n a n d A d v o c a c y (R M A ) … C a u c u s o n N e w a n d U n d e ru s e d R H T e c h n o lo g ie s L A C f o ru m F ra n c o p h o n e fo ru m N o n e o f th e a b o v e Does someone from your organization sit on any of the following: Does someone from your organization sit on any of the following: Does someone from your organization sit on any of the following: Does someone from your organization sit on any of the following: (please specify all that apply) (please specify all that apply) (please specify all that apply) (please specify all that apply) 138 Evaluation of the Reproductive Health Supplies Coalition Perspectives from the Global South (Open-Ended Responses from Q9 from Survey Monkey) Q9. Please give a concrete example or brief story of how being involved with the Coalition has helped you and your organisation. “To be part of the working group has given us opportunities to explore the development of activities/programs with other organizations that are part of the Coalition or to have information about the development of new initiatives by the working groups that could be in favor of our member affiliates.” (INGO) “Through networking, collaboration and share through exchange ideas and information.” (Community based organization) “My organization's participation in the Coalition's activities has opened us to a lot of insights into global level advocacy for family planning and reproductive health. Being involved in the RHSC has improved our ability to attract more funding to implement country level advocacy strategy. For instance, through the RMA-WG, my organization is currently a beneficiary of the Small Grant. It has also enhanced our visibility both at national and global levels, as far as advocacy is concerned. It has also strengthened our credibility and deepened our partnership with in-country key stakeholders in family planning and reproductive health. My organization's participation in the Coalition's activities has opened us to a lot of insights into global level advocacy for family planning and reproductive health. Being involved in the RHSC has improved our ability to attract more funding to implement country level advocacy strategy. For instance, through the RMA- WG, my organization is currently a beneficiary of the Small Grant. It has also enhanced our visibility both at national and global levels, as far as advocacy is concerned. It has also On a weekly basis More than once a … At least once a month At least once a quarter More than once in a … Once per year Less than once per … Inactive 0.0% 10.0% 20.0% 30.0% How would you describe your frequency of participation/activeness in the How would you describe your frequency of participation/activeness in the How would you describe your frequency of participation/activeness in the How would you describe your frequency of participation/activeness in the Coalition? Coalition? Coalition? Coalition? On a weekly basis More than once a month At least once a month At least once a quarter More than once in a year Once per year Less than once per year Inactive 139 Evaluation of the Reproductive Health Supplies Coalition strengthened our credibility and deepened our partnership with in-country key stakeholders in family planning and reproductive health.” (In-country NGO) “As a participant of the LAC Forum we get information of new actions in the field, that helps us to be informed and to ensure we are doing the best practices.” (In-country NGO) “Fund small grants program with RMA Working Group and support variety of other activities, most of which are tied to bringing a developing country perspective.” (Institution of Higher Learning) “My English is not very good, and will write in Spanish. Tuve la oportunidad de participar en una reunion regional y alli pude conocer la amplia discusion técnica y politica sobre suministros. Como parte de este proceso, hemos incluido en el plan de actividades de CLACAI una iniciativa para incorporar a los y las decisores deabogacia en medicamentos e tratar de incidir para que estos incluyan en su agenda el tema de suministro de anticonceptivos y misoprostol. Como resultado de este trabajo, como CLACAI, organizamos una reunion con especialistas es medicamentos en México de 4 países, el mismo que fue realizado a fines del año 2011. (Regional network of individuals and NGOs – see translation below) Translation: "I had the opportunity to be in a regional meeting and there I could know the big political and technical discussion about supplies. As part of this process, we have included in the activities plan of CLACAI an initiative to incorporate the decision makers, Deabogacia, in drugs and try to influence so that they can include in their agendas the supplies of birth control and misoprostol. As a result of this work, we CLACAI, organized a meeting in Mexico, with the specialists in drugs of 4 countries. This was in the end of the year 2011." “To keep in touch with latest information on all the related topics.” (In-country NGO) “We are able to use the RH Interchange to make logistics decisions. The newsletter brings out a lot of lessons to learn from which can be applied to our home situation.” (Government Ministry) “it has helped in sharing challenges met in procuring some of key commodities for RH, including lack of quality assured products for some commodities such as contraceptives. It has contributed to work toward harmonized approach for QA of health products and how to move forward. It provided information and contacts on innovative funding mechanism such as the Pledge Guarantee for Health to prevent treatment disruption and enable better procurement outcomes. It helped in understanding how the Country at Risk (CAR) mechanism put in place to prevent stock out situation for contraceptives and better coordinate donor support for family planning.” (Multilateral Development Agency (including health partnerships)) “Learning about the quality of medicines initiatives supported by the RHSC which has helped our organization plan drug introduction and procurement.” (In-country NGO) “it has to get relevant information and an sight into the nature of our work including providing with the needed resources for running some projects.” (In-country NGO) “Involving my organisation in others initiatives regarding to the family planning and reproductive health field. Sharing also our experience inthis field with coalition members.” (Intergovernment Organization) 140 Evaluation of the Reproductive Health Supplies Coalition “The networks has provided us with recent information and new angles in the areas of reproductive health and family planning and this we have been able to use to localise them for advocacy use and creating awareness among the people in Sub-Saharan Africa through our networks.” (Development Media Organization) “Networking with key opinion leaders in the field of Family planning.” (Pharmaceutical Manufacturer) “the membership of the organization is quite recent but the acquisition of a certificate from the coalition is already adding value to the reproductive health work of NORSAAC. The certificate which is visible in our office makes visitors and partner think NORSAAC is gone international even though we are nationally focused.” (In-country NGO) “Participation in the 2011 Conference resulted in meeting IPAS, an organisation that focusses on abortion services. The interaction has resulted in developing a strong partnership between YONECO and IPAS and currently implementing the SRH and R programmes for young people in the communities.” (In-country NGO) “Up to now, our company had taken part in a meeting in London. We got much information about reproductive health products.” (Pharmaceutical Manufacturer) “In the capacity building experiences, such as Training and Technical Assistance.” (Institution of Higher Learning) “First and most importantly, is the exposure with other international members of the coalition through the meetings, shared materials and the newsletter which we do share frequently. The opportunity to compete for funding though we have never got yet, but is another chance to learn from our shortcomings, as we look for funds to implement our activities.” (In-country NGO) “Being a member of RMA working group strengthened our advocacy for family planning especially when project RMA was stilling running. A strong network of CSOs was formed under RMA and it was coordinated by my organisation a factor that made us achieve a great deal as far as advocacy for RH commodities is concerned. We managed to engage government and increased the budget of FP commodities by almost three folds.” (In-country NGO) “Participated in the LAC Forum in Panama and in the meeting in Addis. Produced a study on risks to contraceptive security in the LAC region. Submitted proposal for small grants to do a study for advocacy.” (Consulting Firm) We simply follow all your e-mailings and news to keep abreast of family planning, reproductive health and maternal health developments, always looking for opportunities in which a very small organization like our own in Venezuela can participate or benefit.” (In-country NGO) “The main example is to have data for edit memoranda for support decision process to contribute to Reproductive Health Organizations.” (Government Ministry) “i am very impressed with the communication style. We are always kept abreast of all relevant information. This has improved our knowledge in RH/FP. Our membership with the coalition has given us some insights into RH and FP activities globally.” (In-country NGO) 141 Evaluation of the Reproductive Health Supplies Coalition Q10. Should the Coalition engage more at country level in order to advance family planning and commodity security? If yes, please check the top three choices that should be prioritized for the Coalition's engagement at country level. If no, please go to the next question. Answer Options Response Percent Response Count Private sector growth for RH commodities 40.7% 22 Supply chain strengthening for RH commodities 50.0% 27 Advocacy to ensure RH security 48.1% 26 Advocacy to increase demand for and access to FP/RH services and supplies 50.0% 27 Resource mobilization and financing for RH commodities 55.6% 30 Assistance to similar in-country groups/networks to promote FP/RH coordination 38.9% 21 Other (please specify) 22.2% 12 answered question 54 skipped question 11 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% P ri v a te s e c to r g ro w th f o r R H c o m m o d it ie s S u p p ly c h a in s tr e n g th e n in g f o r R H c o m m o d it ie s A d v o c a c y t o e n s u re R H s e c u ri ty A d v o c a c y t o in c re a s e d e m a n d fo r a n d a c c e s s t o F P /R H s e rv ic e s a n d s u p p lie s R e s o u rc e m o b ili z a ti o n a n d fi n a n c in g f o r R H c o m m o d it ie s A s s is ta n c e t o s im ila r in -c o u n tr y g ro u p s /n e tw o rk s t o p ro m o te F P /R H c o o rd in a ti o n O th e r (p le a s e s p e c if y ) Country Level Engagement Priorities Country Level Engagement Priorities Country Level Engagement Priorities Country Level Engagement Priorities 142 Evaluation of the Reproductive Health Supplies Coalition The methodology that the evaluators used, in accordance with the SOW, included key informant interviews (in all, over 60 representatives of Coalition members and staff were interviewed) together with extensive document and literature review. In addition, as discussed with donors and included in the Evaluation Work Plan, the evaluators added an electronic survey of all members (65 members responded) and observed/participated, on a more limited basis, in a few Coalition activities, such as Working Group conference calls and meetings. These approaches are detailed below. The concluding section covers data analysis. • Document Review: Primarily during the week of March 5-9, the evaluators reviewed a comprehensive set of RHSC and related contraceptive security (CS) documents. As indicated in the SOW, the RHSC website (www.rhsupplies.org) proved a rich source of materials for review. In addition, the evaluators identified a number of other documents, from both internet searches and some key informants. RHSC documents reviewed include the RHSC Strategic Plan (2007-2015), Terms of Reference (2009), and Monitoring and Evaluation Guidelines and Report; Working Group documents, meeting notes, commissioned studies, articles, published papers; program reports; unpublished technical papers, reports, data compilations, and presentations; video; and multi-media presentations. Notable documents from external sources include “The World We Want – Beyond 2015: A Toolkit for National Deliberations” and “Visualizing the Landscape of Action Networks: An Application of Social Network Analysis”. All documents reviewed for this evaluation are listed in Appendix C of this Report. • Interview Key Informants: For purposes of this evaluation, a key informant was defined as one with breadth as well as depth of knowledge about RHSC and/or the broader field of CS. The SOW specified interviewing at least 45 key informants. In all, the evaluators interviewed 62 key informants from 36 organizations. As the SOW encouraged, most of these interviews were done in-person. Some key informants were interviewed by phone, particularly in the Global South countries and regions where RHSC partners are active. A few key informants were not directly connected with RHSC but offered broad perspectives on this collaborative approach, alternative models, and/or commodity security. Working from a preliminary list developed during the March 15th meeting in London, key informants interviewed included the Executive Committee Chair and members, donor group representatives, the Coalition director and staff of the RHSC Secretariat, and representatives of other organizations who have current or previous involvement with RHSC. The evaluators made particular effort to interview current heads of the three Working Groups as well as a selection of participating members. The evaluators also gave some special attention to some donors, particularly ones who benefit from and may potentially contribute to RHSC-like operations in the future. The list of all those interviewed is in Appendix _ of this Report. To interview key informants, the evaluators developed semi-structured questionnaires using open-ended questions. Building upon the three SOW objectives and 19 associated key questions, the evaluators generated a generic questionnaire. This questionnaire was framed Annex 6 Evaluation Methodology and Data Analysis 143 Evaluation of the Reproductive Health Supplies Coalition within the eight overarching questions. In advance of each key informant interview, the evaluators developed a tailored questionnaire. The generic questionnaire, which provided the base from which questions were selected for these interviews, is appended to this Report as Appendix _. Most in-person interviews occurred between March 21st to April 4th, when the evaluators were in Europe, and April 30th to May 11th, when they were on the East Coast of the US. Residential proximity enabled the evaluators to have face-to-face interviews with some key informants in the UK and on the West Coast of the US. For others, notably those working within the countries and regions where RHSC partners are active, the evaluators conducted interviews by skype or phone. Some selected key informants were interviewed more than once. This was particularly the case with the RHSC director who made himself available for multiple interviews over several days. In addition to being interviewed as individual key informants, donor representatives also had, opportunities to guide this evaluation during the meetings of March 15th and May 11th. The list of key informants, from which the evaluators identified and conducted interviews, is in Appendix _. • Survey of RHSC Members: To ensure that the entire membership of RHSC could input into this evaluation directly, all member organizations were given the opportunity of completing an electronic survey. This survey was not included in the original SOW but was added as a useful adjunct to the evaluation after discussion with the donors. A short-term consultant was engaged to assist with the administration, tabulation, and analysis of survey results. Using Survey Monkey, the evaluators designed a short questionnaire, which was subsequently emailed to all member organizations in late April. The survey included a number of closed questions as well as a limited number of open questions. Based on analysis by the short-term consultant, the evaluators subsequently interpreted the data that resulted and incorporated these findings into the evaluation. • Participation/Observation: On a limited basis and as opportunity and time allowed, the evaluators listened in on Working Group conference calls and meetings. On one occasion, the June 4-5 MDA Working Group meeting in California, enabled an evaluator to participate. In addition, during visits to Brussels in late March, the evaluators observed working conditions and operations within the Secretariat at first hand. During this visit, as well as during subsequent interactions with staff at PATH, the evaluators also asked for assessments of and requested updates on the Monitoring and Evaluation indicators. Though limited by time constraints, such opportunities enabled the evaluators to collect a different level of data than that yielded by interviews or literature review. Data Analysis: Documents reviewed and key informants responses to interview questions are the primary data sources for this evaluation. These data are complemented by responses received from the electronic survey of all member organizations. In addition, the evaluators deepened understanding of the Coalition through opportunities for participation/observation of RHSC operations and activities. 144 Evaluation of the Reproductive Health Supplies Coalition To access and organize information from the literature review, documents reviewed were assessed by relevance to the various questions included in the SOW. This enabled interviewers to reference documents as appropriate to the great variety of subject area inquiries put forth in the SOW. By collating SOW questions in terms of eight overarching questions that frame this evaluation, the evaluators were able to extract from this broad review of documents salient information for formulating answers to questions at various levels. A similar process of codifying and layering questions was used to structure key informant interviews and analysis of responses. Almost all interviews were digitally recorded (voice recordings) and saved as electronic files. The recordings were then transcribed. To analyze key informant responses, the evaluators also reviewed responses by listening to the digital files and reviewing the written transcriptions. Interview questions were coded in relation to the three Evaluation objectives, the 19 associated key questions in the SOW, and the eight overarching questions that frame this evaluation. This feature of the interview data set made possible collation of responses at these three different levels. While this facilitated analysis of some responses on specific topics, within time constraints it was not possible to comprehensively analyze all responses at these different levels. Stored as electronic files, selected content areas within these files were also searched for particular themes and issues relevant to the evaluation. Analysis of data proceeded at three levels: the 19 key questions in the SOW; the eight overarching questions; and the three SOW objectives for this evaluation. Responses to the electronic member survey, though far less comprehensive than the key informant responses, were also reviewed in relation to the eight overarching questions. Preliminary summations in terms of SOW and overarching questions were checked against information from project documents, member survey responses, and opportunities to listen to or observe RHSC Working Group meetings. Triangulation occurred in terms of what documents indicate RHSC intended to and/or did accomplish, what key informants reported about RHSC, and what the evaluators learned from the combination of document review, interviews, the web-based member survey, and occasions to observe RHSC in action. As encouraged by the donor group, special attention was paid to future prospects and potential for collaboration among partner organizations to advance FP/RH commodity security, as well as family planning more broadly, especially within mid and low income countries. A meeting held on May 11th with the donor group provided opportunity to present and discuss preliminary findings emerging from this analysis of data. Goal 1: Increase availability, predictability, and sustainability of financing for RH Supplies. Objective Indicator level Summary of results Progress Objective 1.A: Increase flow of new resources. 1A.1 Number of focus countries with a national budget line for RH supplies Country Completed by 2009 + 1A.2 Percentage increase in donor funding for contraceptives in focus countries Global Increased occurred but volatile: Donor funding in 14 focus countries increased by almost 67% from 2006 – 2001, but significant variations year on year. +/- 1A.3 Percentage increase in donor funding for contraceptives worldwide Global Donor funding worldwide increased from 133m to 219m between 2006 – 2011 (approx 65%). Remained static or fallen between 2009 and 2011. +/- 1A.4 Increase in number of donors making first-time contributions for RH supplies Global Since 2009 four donors have made first time contributions to the GPRHSC. 1A.5 Number of GFATM country proposals per year with provisions for the purchase of RH supplies Country Number of countries procuring RH supplies with GF resources has increased from 6 (2006) to 20 (2010). + 1A.6 Increase in level of funding allocated to RH supplies by GFATM Global From a baseline in 2006 of $800k funding rose to $10.4m in 2009. Since then it has declined to $2.6m in 2011. +/- Objective 1.B: Ensure that countries and their governments make use of the resources already at their disposal. 1B.1 Proportion of focus country budget line(s) for RH supplies actually spent Country Of 13 focus countries, 10 spent 100% or more of budgeted expenditure on contraceptives in 2010 ++ 1B.2 Examples of new domestic financing mechanisms for supplies proposed or introduced in focus countries Country No new mechanisms implemented or planned - Objective 1.C: Mobilize Total Market Resources 1C.1. Examples of new initiatives undertaken in focus countries to expand the total market for RH supplies Country 2 total market initiatives undertaken. Substantive impact not yet clear. +/- 1C.2. Increase in the non-public sector share of the market for condoms, pills, IUDs, and injectables in focus countries Country In the four focus countries currently measurable, only Nepal shows increase in non-public market share. - Annex 7 Summary of results against the MEF 146 Evaluation of the Reproductive Health Supplies Coalition Goal 2. Strengthen capacity of health systems to deliver RH supplies in a sustainable manner Objective Indicator level Summary of results Progress Objective 2.A: Ensure the integrity of systems for supply chain management, both globally and at country level. 2A.1 Number of focus countries with functioning RH commodity committees or similar mechanisms for managing RH supply chains Country All 14 focus countries achieved this by 2009. +++ 2A.2 Number of focus countries that include RH commodities on the national essential drug list Country All 14 focus countries achieved this by 2009. Since 2009 types of commodities on EDLs have now doubled. +++ 2A.3 Examples of formal, coordinated actions by focus countries to improve RH supply chain performance (e.g. SPARHCS, national needs assessments, etc) Country 11 focus countries have undertaken such efforts. Substantive impact on supply chain performance not reported. +/- 2A.4 Number of RH products from specific producers/factories, pre-qualified under WHO/UNFPA Programme Global Between 2007 and 2010 33 more products were pre-qualified. + 2A.5 Number of all countries (and focus countries), per year, with active registered users of the RH Interchange Country All 14 focus countries now registered. 92 counties in total have registered users. 632 users registered in total + 2A.6 Decline in stock out rates at public-sector service delivery points in focus countries Country Mozambique and Ethiopia show improvements. Rwanda & Nepal performance remains high. Burkina Faso and Mongolia worsening. +/- 2A.7 Descriptions of private sector supply chains for RH supplies in the private sector and use in the public sector Country One joint project being carried out by Bayer and USAID in sub-Saharan Africa. + 2A.8 Amount of funding provided through mechanisms that make RH supply resources less volatile Global AccessRH: Since its start in 2011(?), the value of orders has totalled $32m. $12m mobilised for implants for Ethiopia. ++ Objective 2.B: Provide a coordinated, short-term response to stock outs, supply disruptions and other failures that occasionally befall functional supply chains. Indicator 2B.1 Description of stock-outs averted or resolved through the intervention of the Coordinated Assistance for RH Supplies (CARhs) Group Country In 2011 the CARhs group successfully resolved 61% of all cases requiring direct intervention. ++ Objective 2.C: Promote effective delivery of RH health supplies in settings where routine systems for supply chain management have broken down or ceased functioning. Indicator 2C.1 Number of Emergency RH kits sent to different countries by humanitarian agencies per year Country UNFPA dispatched over 47,000 RH emergency kits between 2009-11. + Indicator 2C.2 Examples of Coalition-inspired initiatives to integrate RH supplies into humanitarian relief efforts Country Changes in MSI operations and strategic planning for RAISE initiative prompted by one such initiative. +/- 147 Evaluation of the Reproductive Health Supplies Coalition Goal 3: Assure the added value of the Coalition as a productive and sustainable global partnership through efficiency, advocacy and innovation Objective Indicator level Summary of results Progress Objective 3A. Foster greater harmonization among stakeholders at global and country level to generate economies of scale, maximize impacts, and minimize the risks of redundancy and confusion. 3A.1 Examples of coordinated action by any two or more Coalition members, in the same focus country, with the explicit aim of strengthening RH commodity security Coalition Coalition members worked together in at least 6 countries since 2009: Ethiopia (Implanon initiative); Honduras (total market initiative); Mexico (research on investments in FP); Nicaragua (needs assessment on HR capacity in supply chain management); Tanzania (pooled resources for FP and MH); Uganda (advocacy for increased funding to FP) + 3A.2 Examples of coordinated action by two or more Coalition institutions with the explicit aim of strengthening RH commodity security beyond the boundaries of a single country. Coalition RHI, CARhs, ‘People that Deliver’, formation of LAC & SECONAF. ++ 3A.3 Examples of new Coalition tools and/or publications aimed at comparing or harmonizing comparable activities being implemented by 2+ Coalition members Coalition HAND-to-HAND campaign + Objective 3B. Generate, disseminate, and facilitate the application of knowledge through innovation, research, and documentation. 3B.1 Examples of all new tools, publications, and research initiatives produced each year, either by the Coalition or by its members in response to Coalition inspired initiatives Coalition On-line reference library; country profiles; LAPTOP; Innovation Fund; ++ 3B.2 Annual increase in visits to the Coalition’s website Coalition Monthly average visits 547 in 2008; 1600 in 2009 ; 1738 between June 2011 – March 2012 ++ Objective 3C. Offer a credible and sustainable institutional framework within which stakeholders can collaborate, build commitment to supply security, formulate strategies for change, develop common indicators of success, and measure progress against them. 3C.1 Completion of Monitoring reports in 2009 and 2015 Coalition Achieved in 2009. Additional report produced for 2012 evaluation. ++ 3C.2 Number of institutions that contribute financially, each year, to the implementation of Coalition-sponsored or -inspired activities (excluding core support to Secretariat operations) Coalition Variable numbers of contributors. highest: 2007 (21); lowest: 2011 (7) but this year had the highest volume of financial support. + 3C.3 Total financial contribution of all institutions, per year, to the implementation of Coalition-sponsored or - inspired activities Coalition Funds leveraged; $1.28m in 2007; $7.92m in 2009; $25m in 2011 including added value from savings in implant prices. ++ 3C.4 Total financial contribution of all institutions, per year, to sustaining Secretariat’s core operating costs Coalition Total contribution to core costs: $853k in 2006; $1.19m in 1009; $2.08m in 2011 +++ 148 Evaluation of the Reproductive Health Supplies Coalition Objective Indicator level Summary of results Progress 3C.5 Total number of Coalition members and increase Coalition Total numbers of members: approx 50 in Feb 08; Approx 105 in Feb 10; 159 at end of 2011. +++ 149 Evaluation of the Reproductive Health Supplies Coalition Oct 2010 – Sept 2011 Action outcomes Existing Stockouts Avert Future Stockouts Existing Overstocks Other (stocks btw max/min) Action Outcome Totals CARhs action not needed 3 16 3 2 24 CARhs unable to act 3 7 0 0 10 CARhs only able to provide info 4 1 0 1 6 CARhs took action that did not add value 0 0 0 0 0 CARhs took action that added value 6 14 4 1 25 Issue Type Totals 16 38 7 4 65 Source: Coordinated Assistance for Reproductive Health Supplies Group, Progress Report: October 2010 – September 2011, page 3 Annex 8 Outcome of CARhs engagement. 150 Evaluation of the Reproductive Health Supplies Coalition Country level uptake and results Activity Purpose Uptake and results at country level Total Market Initiative in Honduras. Facilitate collaboration among providers of family planning products and services and segment the family planning market to better meet the needs of clients. New connections established and broad-based local partnerships built across public and private sectors. Secured the Ministry of Health’s leadership of a total market approach and overcame mistrust of the private sector. Re-establishment of the Contraceptive Security Committee, which has since started coordinating new efforts in contraceptive financing, procurement, and distribution Total Market Initiative in Madagascar. To build public sector support for a total market approach by demonstrating how greater involvement of the private sector can increase contraceptive prevalence and benefit the country’s vulnerable and poor populations. Recommendations adopted by the Ministry of Health. Political ownership forms the basis for continued engagement and the development of joint strategies and guidelines. Professional development opportunities for logistics and supply chain managers: phases 1 and 2 Improve skills in supply chain management by creating an online directory of English, French, and Spanish-language professional development training programs for public- sector supply chain managers in developing countries. ‘Laptop’ recently launched on the SC website. Potentially useful. Too early to judge results at country level. Assessment of logistics systems in three crisis settings. Develop a better understanding of the challenges facing agencies that deliver emergency RH supplies Grant Implementing agencies have installed logistics coordinators at two sites and add logistics management to the job description of a program manager at a third site. MSI now providing reproductive health commodities to its partners in the study countries. Partnership formed between the International Dispensary Association and MSI to include comprehensive commodities on the standard International Dispensary Association list of emergency essential medical supplies. Findings were also incorporated into RAISE initiative. Two-year awareness raising of supply security in francophone Africa. Build commitment towards RH supplies among decision makers, NGOs, parliamentarians, and the media. Toolkit on reproductive health supplies translated, interest of donors and stakeholders built, NGOs supported to carry out advocacy in Niger, Burkina Faso, Benin and Mali. An important grounding for the future work of the SC in this region. Annex 9 Country relevant Innovation Fund grants 151 Evaluation of the Reproductive Health Supplies Coalition Activity Purpose Uptake and results at country level Identify, develop, and disseminate best practices in supply chain management for community based distribution programs Increase access to family planning products and services. Programme manager’s guideline for design of supply chain management systems in community distribution programs developed. Recommendations swiftly applied in Ethiopia. Raise awareness of the WHO Prequalification Program and calculate the potential benefits to manufacturers of becoming pre-qualified Identify hormonal contraceptive manufacturers with the best prospects for achieving WHO prequalification in the shortest timeframe and help them understand the actions, level of investment, and time requirements needed to do so. Established a roster of pre-qualified hormonal contraceptive manufacturers with the capacity to acquire market authorization. Raised understanding and awareness of the WHO prequalification programme and its benefits for reproductive health supplies manufacturers. Identified three key areas for further work: manufacturer compliance, bioequivalence, and the quality of active pharmaceutical ingredients. Important step to increasing availability of lower price generics for countries. 152 Evaluation of the Reproductive Health Supplies Coalition CARhs RHI AccessRH PGH Initiation Established under the auspices of Coalition in 2005. Was established in parallel to the creation of the Coalition, with Gates funding. The mechanism was developed as a consequence of supply chain bottlenecks identified by Coalition-commissioned research (Mercer report) The mechanism was developed as a consequence of supply chain bottlenecks identified by Coalition-commissioned research (Mercer report) Funding DFID support ($10 million) to the CARhs emerged from discussions among key Coalition members. In 2009, Coalition (esp. SSWG) provided RHI with a forum for discussions with donors over long-term funding. Coalition played a direct role in funding 1) the development of AccessRH and 2) the launch of the AccessRH mechanism. Its first two large donor grants were arranged by Wolfgang Bichman, Coalition Chair; and Antoinette Gosses, EC rep on Executive Committee member. Coalition also funded early feasibility study on manufacturer engagement under the Innovation Fund Coalition played a direct role in funding 1) the development of PGH and 2) the launch of the PGH mechanism. Its initial funding grant came from Gates Foundation Coalition has covered certain operating expenses of PGH since end of Gates funding Facilitating Implementation From 2007 to 2009, the Coalition Secretariat coordinated meetings of the CARhs, scheduled meetings. Responsibility was then transferred to USAID in 2009(?) and UNFPA in 2011. Coalition was a member of RHI advisory board until the board was subsumed under the AccessRH board. Four of the 14 Coalition focus countries were chosen to meet RHI outreach needs RHI is on Coalition URL (rhsupplies.org) Coalition hosted WG discussions leading to development and launch of AccessRH. Served as bridge for new products in AccessRH inventory (implants). Through its participation in the Coalition-hosted manufacturer stakeholder groups, AccessRH Ensure engagement in financing affairs Set up meetings with Finance and other ministers Financed and supported participation of PGH at conferences. Annex 10 Contributions of the Supplies Coalition to Key Initiatives 153 Evaluation of the Reproductive Health Supplies Coalition CARhs RHI AccessRH PGH was able to shave $1 off the price of Jadelle and .005 cents off price of FC2. Scope of Initiative Provided continuity for idea Served as a vehicle by which Coalition members could keep PGH in limelight, despite end of Gates grant. Content/ Technical rigor CARhs submits its annual report each year to the SSWG for review & discussion Institutional memory provided by Coalition members on AccessRH’s Advisory committee (AC) ensures that it stays aligned with its original objectives. Coalition members on the AC have enabled ARH to overcome hurdles operating within the UN system (ie. Letter to Babatunde) Visibility/ Advocacy PPMR to be hosted on Coalition website Monthly updates on CARhs reported in SupplyInsider Spread the word to encourage users Two-thirds of all RHI hits are though Coalition website RHI has used Coalition and WGs to disseminate information on RHI and how to use new features as they are developed (graphics, data analysis, etc.) Access RH widely featured in Coalition communications materials (eg. Website homepage) Linkage with Coalition allows AccessRH to be seen as more than just UNFPA procurement. Together with PGH, played key role in opening up the possibility of UNFPA securing credit for AccessRH procurement (2011 Jadelle deal) Announce deals though SupplyInsider, website and other conduits Sustains PGH engagement on all Coalition issues relating to manufacturers, pricing, etc. 154 Evaluation of the Reproductive Health Supplies Coalition HANDtoHAND LAPTOP Support for QA of supplies, including role of WHO Prequalification Initiation HANDtoHAND (H2H) was conceived and developed by the Coalition as a framework for channeling the support of the RH community towards the Secretary General’s Global Strategy for Women and Children’s Health After presenting the results of an initial $67,000 Innovation Fund award (which consolidate SCM course openings on an Excel spreadsheet), the Coalition recommended that JSI expand the effort by developing a user-friendly on-line tool; and encouraged them to apply for a follow-on IF grant ($122,000). WHO Prequal was not a Coalition initiative, unlike the others. But the Coalition has, from early on, played a key role in bridging the initiative and the broader supplies community. The Prequal was a workstream of the MDA WG going back to 2005. Breadth of Engagement Coalition has been involved in every aspect of the Campaign: from development and launch; to the solicitation of commitments; to dissemination of information; to more recent efforts to strategize implements and now – coexistence with the new FP Summit Coalition has served as a forum for member engagement with the WHO Prequal process (going back to 2006 meeting in Bonn) H. Hogerziel has always used Coalition meetings to update broader community on progress under PQ Funding Apart from the hundreds of person hours invested by Coalition and Secretariat staff; the Coalition contracted the communications firm, GMMB (approximately $75,000) to develop key messages associated with the launch. Coalition invested nearly $190,000 in development of tool via Innovation Fund. Coalition continues to finance technical backstopping by our web-manager, Active Elements. Coalition introduced possibility of homepage advertising and secured all ads displayed to date. Facilitating Implementation As noted above, the Coalition has been involved in every aspect of the Campaign. It has played an especially central role in the dissemination of information (underpinnings of the Campaign; member commitments, announcement of developments) vis the website and social media. The Coalition also hosted in September 2011, a large workshop designed to strategize implementation of the campaign. The workshop was held in NY and attended by 25 participants from across the Coalition. Coalition secured the engagement of People that Deliver to co-manage the website. Since the end of the Innovation Fund grant, the Coalition continues to serve as the overall “management umbrella” under which key partners (PtD and now IAPHL) will continue to work. Coalition served as conduit for communicating member-requests for product calls (2 rod implant) 155 Evaluation of the Reproductive Health Supplies Coalition HANDtoHAND LAPTOP Support for QA of supplies, including role of WHO Prequalification Scope of Initiative The Director believed that involvement by the Coalition turned what might otherwise have been perceived as a “call for contributions” (albeit for a common cause) into a joint movement - owned by those who have committed. We cultivated that sense of “partnership” by hosting special events for H2H partners (eg. the dinner at Access for All) and by targeting messages to them; highlighting contributors in SupplyInsider, etc. The Director’s view is that involvement by the Coalition has broadened the scope of the tool – from what began life largely as a “technical exercise” to a vehicle for adding value to the work of members; and for bridging partners through advertising; the award of scholarships, etc. enabled JSI to Now under QuRHM, Coalition is becoming the nexus for discussions on Pre-Qual, ERP, etc. Content/ Technical rigor Coalition played a central role in the formulation of the 100 metric. Under the leadership of USAID, the Coalition participated in meetings with key demographers; Coalition was also represented more recently in the development of the FP Summit metric (120 by 20). Coalition has carefully reviewed all front-end applications of the web-based tool; and submitted recommendations to JSI and AE. Coalition has served as the principle conduit for funding technical support on behalf of WHO Prequal: 1: Innovation Fund $200,000 to Concept; 2. DFID funding $5 million to Concept via the Coalition Visibility/ Advocacy Coalition has been the primary conduit for dissemination of information on H2H through its website, news flashes and social media. The relationship forged with GMMB in the early days of the campaign, continued on through the implementation of the Access for All conference. Coalition has been the primary conduit for dissemination of LAPTOP through its website, news flashes, and social media. Coalition has also been central in the development of all messaging and outreach surrounding the new tool. Coalition introduced (and financed) a small scholarship fund, in part to heighten visibility of LAPTOP and generate traffic to it. LAPTOP is currently the most frequently visited page on the Coalition website. Latest Prequal product lists are posted on website. All new prequalifications are announced on website. As Coalition, we are free to announce brand names of prequalified products to ensure greater visibility. Added value for individual members Individual members value the coalition because it brings them: • The ability to reach a broad and intricate group of organizations in the field. • The opportunity to form new partnerships. • Access to funding from other members. • The opportunity to share their lessons learned and benefit from others’. • A platform for discussing issues which are important to them • The opportunity to further their advocacy efforts. For example one international NGO reported that “We have worked via the structure of the RHSC to further all of our advocacy objectives -- for example by using the working group structure managed by the RHSC to build a community of advocates and country NGOs active on the issue”.11 • Access to information on initiatives, best practice, experience and what other partners are doing and thinking. • A better understanding of complex technical issues. • A sense of shared responsibility and mutual accountability • Tools and strategies to help some organizations raise awareness of the family planning agenda within their own institution. • Seedcorn financing to initiate new ideas that cannot be funded elsewhere. Added value for country level efforts The direct added value for country level efforts is limited given that the coalition has very limited reach into countries, mostly through international member agencies with local partners. Some of these global level members were skeptical about the direct added value of the coalition to their work at country level. However others, and some members based in country such as national NGOs were able to identify benefits that impacted on their ability to work effectively at country level. These included: Added value for country level initiatives is drawn from • Access to resources to run projects. • Access to new initiatives that can be used at country level. • Information that helps them to ensure they are following best practice. • Direct access to key country officials though meetings, which has helped resolve problems and bottlenecks. One pharmaceutical manufacturer reported: “Being involved in the Coalition helped us resolve a long pending issue related to a Sub-Saharan country, where generics were unable to make any headway despite all efforts. In the PPD meeting in India around 2005, we could meet the Minister of Health of the country along with UNFPA Procurement Head and actually resolve the issue.”12 • The use of RHI to inform logistics decisions. _________________________ 11 Member survey p6 12 Ibid p4 Annex 11 Added Value of the Supplies Coalition 157 Evaluation of the Reproductive Health Supplies Coalition • Access to lessons for countries, through the newsletter. • Catalyzing country level work. A member from a private Foundation observed “The coalition held a Webinar on contraceptive stock outs and galvanized 80 members to provide feedback on a Gates Foundation assessment that increased the dialogue on this critical issue and launched new work in this area in Nigeria and Senegal.”13 • Forming links to global networks. • Forming links with members in other part of the world. • Providing an insight into the work of other members. Added value for regional efforts According to the member of the LAC forum who was interviewed, the added value of the regional coalition is that it is: • Enabling the development of joint proposals between countries which is difficult to do without the ‘glue’ of LAC. • Providing a more solid base for lobbying governments. • Helping to overcome the language barrier by translating tools and contextualizing them. • Providing an opportunity at the global meeting to meet members of the wider coalition and articulate their need for support. The francophone coalition SECONAF is not yet able to demonstrate added value, but the global Coalition was of assistance as Equilbre et Populations (EP) carried out some initial work in West Africa with Innovation Fund support. The Coalition added value to their efforts by: • Helping to identify organizations or agencies that EP could work with such as UNFPA. • Enabling the translation of the advocacy toolkit into French which was a very important contribution to forming links with partners and building their capacity. • Enabling access to other tools and information through the website. • Enabling access to other partners and ideas through the annual meeting. • Continuing and strengthening existing relations. _________________________ 13 Ibid p5 158 Evaluation of the Reproductive Health Supplies Coalition Reproductive Health Supplies Coalition External Evaluation: March-June 2012 RHSC Evaluation Executive Committee Meeting June 25-26, 2012 What is RHSC now? What could the Coalition become? What would donors likely support? What does RHSC need to do to more fully impact FP/RH commodity security? RHSC within FP/RH STRATEGY Annex 12 Presentation to Executive Committee, June 25th 2012 159 Evaluation of the Reproductive Health Supplies Coalition EVALUATION S.O.W. • ACHIEVEMENT Assess how well the RHSC has met its objectives and made an impact on RH commodity security • STRUCTURE/PROCESSES Assess how well RHSC structures and processes have worked towards achievement of RHSC objectives • FUTURE DIRECTIONS Recommend how the RHSC’s role should evolve in light of the current and emerging environment, and identify measures to improve the efficiency and effectiveness of the RHSC 3 As detailed in the Strategic Plan, the RHSC’s vision is for: “all people in lower- and middle income countries to be able to access and use affordable, high-quality supplies, including a broad choice of contraceptives, to ensure their better reproductive health.” RHSC Evaluation S.O.W., March 2012 EVALUATION METHODOLOGY •Document review • Key informant interview • Members’ survey • Participation / observation A two-person team will conduct the evaluation. The evaluation team will use a variety of methods for collecting, analyzing, and synthesizing data and information. The following essential elements will be included in the methodology, as well as any additional methods proposed by the team…review pertinent documents, reports, and other materials… conduct qualitative, in-depth interviews with key stakeholders and partners. RHSC Evaluation S.O.W., March 2012 160 Evaluation of the Reproductive Health Supplies Coalition MARCH APRIL MAY JUNE Document Review London Interviews Telephone, West & East Coast Interviews Europe Interviews Analyze & Synthesize Donors’ Review 2012 Report drafting !- . . . RHSC EVALUATION TIMELINE Executive Committee meeting Visit to Secretariat Donor meeting in London Workplan accepted Report submission Donor update Donor response Report revision Members Survey WG Meetings S.O.W.: ACHIEVEMENTS? STRUCTURE/PROCESS? FUTURE? EIGHT OVERARCHING QUESTIONS TWENTY-ONE KEY S.O.W. QUESTIONS 89 GENERIC INTERVIEW QUESTIONS 64 INDIVIDUALS INTERVIEWED 6 KEY INFORMANT INTERVIEWS 161 Evaluation of the Reproductive Health Supplies Coalition 2004-07 2007-2008 2009-2010 2011-2012 Interim WG on RH Commodity Security Gates Foundation funds RHSC Core Group of Donors and INGOs remain involved Innovation Fund Starts Uganda Meeting Collaborative Donor Support for RHSC: DFID, Gates, UNFPA, USAID, !- . . . EVOLUTION of RHSC Executive Committee Meeting Secretariat Established Seattle RHSC Meeting Working Groups Formed Addis Meeting Membership Criteria Opens Up RHSC Evaluation SOW RHSC Evaluation IF Awards IF Awards IF Awards 1. What impact has RHSC made on improving commodity security at global, regional, and country levels? Global impact clearly demonstrated • Increase in funding • Prominence of FP/RH commodity security has increased • Greater profile of FP/RH commodity security on the global health agenda. • Better donor coordination • Effective tools have been nurtured and / or encouraged eg CARhs, PGH Regional impact is becoming apparent • Demonstrated through LAC • More expected through SECONAF Limited country level impact • Ethiopia (Implanon) • Through Working Groups (e.g. RMA, TMI) 162 Evaluation of the Reproductive Health Supplies Coalition 2. Over and above partner organizations’ own accomplishments, what is RHSC’s added-value and comparative advantage (value-for-money)? Comparative advantage lies primarily in the Coalition’s neutrality, reach, and resulting power as a convener Added value varies for type of member but evident for all Good value for money: • Efficient and cost effective secretariat • Effective outputs of Coalition • Demonstrable leveraging power 3. To what extent is RHSC’s role adequately reflected in its theory of change and M&E framework? 2007-15 Strategy • Has served well • However expanded membership, current conditions, and future possibilities necessitate revision There is no theory of change M&E framework needs a major overhaul • Does not fully capture what the Coalition has or could hope to achieve • Not used as a management tool • Not used to promote accountability 163 Evaluation of the Reproductive Health Supplies Coalition 4. In what ways and to what effect has the structure and operation of RHSC evolved, particularly since 2009? • Current structure: Terms of Reference (2009) • Executive Committee &Secretariat • Working Groups & Members • Flexibility and Adaptations • Caucus for New and Underutilized Methods • Task Force for Maternal Health Supplies • Regional Forums -- LAC and SECONAF • 50% of new members from within countries • Support for Structures and Operations • Secretariat Staff Changes and New Staff Positions • Working Group Leader Turnover and Secretariat Support • PATH continues back-office support for Secretariat 5. What works well and what needs to be improved or changed about RHSC’s internal and external operations? Strengths within RHSC Internal Operations • Strong Leadership and Supportive Secretariat • Volunteerism and Commitment • Working Environment and Communications: across three locations • Highly Committed and Supportive Donors Weaknesses within RHSC Internal Operations • Secretariat Staff -- Overworked • Volunteerism has limits • Leadership / facilitator / convener model – not always clear • Working Groups as “engines” – model outdated and/or needs update • Absorbing Rapid Membership Increase External Operations • Growing Stature of RHSC • Changes in Commodity Playing Field (manufacturer engagement, global data and procurement approaches, influenced UN Commission on Live Saving Commodities) • Limited Reach into Countries 164 Evaluation of the Reproductive Health Supplies Coalition Breadth and Depth of RHSC Procurers Suppliers Providers • Donors • Govts. • Pharma • INGOs • Regional • CountryD e pt h a lo n g S u pp ly C h a in R e a c h a c ro s s S u p p l y C h a in 6. To what extent has RHSC been responsive to partner-led initiatives or proactive towards new opportunities? Partner-Led • Working Groups depend on member contributions -- “Coalition of the willing” • Innovation Fund – studies, activities, and projects emerged from Working Groups • Caucus and Task Force emerged from member interests • Regional Forums: avenues to advance FP/RH and CS in countries at different levels Proactive RHSC • Hand-to-Hand • Manufacturers Upsides/Downsides • Well Funded and Aligned Members = most active partners • Country-level Initiatives minimal • Top-down vs. Bottom-up Leadership • Coalition that embraces both could prove powerful 165 Evaluation of the Reproductive Health Supplies Coalition 7. Does RHSC seem poised and/or have potential for improving and increasing contraceptive security in the future? Solid core of member organizations to build on RHSC well regarded and increasingly recognized Membership roles likely to continue increasing – adding 20 per month Spans large array of stakeholders Future depends on: • Continuing member commitment and involvement • Funding to support Secretariat and value-added contributions • Re-focusing and re-strategizing to move impact further down supply chain 8. Beyond serving as an umbrella for collaboration, what alternative models or approaches could and should the RHSC adopt that are needed in the coming years to advance RH commodity security globally, regionally, and at country level? Retain core focus on RH commodities Maintain global as well as local focus on supply chain improvement Promote greater country level ownership of the Coalition Retain but supplement the partner-centric funding model Step up to the RHSC role as a leader in the field Create a closer relationship with UNFPA’s GPRHCS Continue to encourage regional forums, with provisos Revisit the Strategic Plan Preserve the comparative advantage of the RHSC 166 Evaluation of the Reproductive Health Supplies Coalition • Maintain core focus RH commodities • RHSC: lean & mean trim & more technical • Value for money: keep high value, stay well positioned • Develop theory of change , update strategy, and revise M&E framework • Country Participation: Minimal moderate • Regional Advances: Promising but Limited • Innovation Fund Catalyst Fund • Donor coordinated funding for RHCS = increased / improved donor collaboration 18 MAJOR RECOMMENDATIONS COMMUNICATIONS CONVENING • Secretariat as node • Website – lags behind • Newsletter leads • Interactivity potential • Members as facilitators • Professionalize facilitation • Make more of meeting methodologies CORE FUNCTIONS 167 Evaluation of the Reproductive Health Supplies Coalition COLLABORATION NETWORKING • Working Groups as core structure • Cross-Group connections • Incentivize • Connect into countries • Connect across countries • Social Networking �Mapping � Tracking �Analysis CORE FUNCTIONS 168 Evaluation of the Reproductive Health Supplies Coalition CONSIDERATIONS and QUESTIONS • Working Group structure too limiting? • Should / could countries become the engines of RHSC? • How much engagement beyond commodities, especially in-country? • How many more and what skill set for Secretariat staff? • When and how to update the Strategic Plan? • What role for the RHSC following on the Family Planning Summit? • What is the RHSC doing well and should continue doing? • What should RHSC adjust or change to do better? • What should the RHSC do much better or greatly different from what it has done in the past? 169 Evaluation of the Reproductive Health Supplies Coalition Annex 13 Reproductive Health Supplies Coalition. Monitoring Report (2009-2011) Reproductive Health Supplies Coalition Monitoring Report (2009-2011) 170 Evaluation of the Reproductive Health Supplies Coalition Introduction The Reproductive Health Supplies Coalition (the Coalition) is a partnership of governmental, non-governmental, public, and private-sector agencies dedicated to ensuring that people in low and middle income (LMI) countries can obtain and use high-quality reproductive health (RH) supplies, including a broad range of contraceptive methods. Securing RH supplies is essential to the effective provision of RH care, which underpins the meeting of the Millennium Development Goals (MDGs). In three years, the Coalition has grown from a small grouping of like-minded individuals to a structured partnership of key institutions in the RH supplies field. In October 2005, the Coalition approved its “Organizational Principles and Structural/Governance Terms of Reference” (TOR). Soon thereafter, it drafted a strategic plan that consolidated into one document its vision, mission, guiding principles, goals, and objectives. The Coalition’s Monitoring Framework is directly linked to goals and specific objectives outlined in the Coalition’s Strategic Plan—a linkage that has profound implications for the Monitoring Framework itself and for the way in which data is collected. Like the Strategic Plan, the Framework monitors changes at multiple levels: globally, at country level, and at the level of the Coalition itself. Secondly, implementation of the Monitoring Framework, like that of the Strategic Plan, relies on data collected and submitted by member organizations. And thirdly, in addition to comprising indicators whose outcomes remain fully dependent on the Coalition (such as completion of Secretariat milestones), other indicators remain more dependent on the collective work of Coalition members as a whole. Given its linkage to the Coalition’s Strategic Plan, the Monitoring Framework assesses the Coalition’s success by tracking 30 indicators linked to a subset of activities originally proposed in the Plan. Seventeen of the indicators are quantitative in nature and measure the impact of Coalition activities at the country/regional-, global-, and Coalition-level. Thirteen are qualitative in nature. The Monitoring Framework also establishes a baseline (January 2007) and two targets—one for 2009 (to coincide with the end of the first tranche of core funding), and the other for 2015 (endpoint of the Coalition’s current strategic framework). Routine monitoring of certain key indicators has been taking place since the adoption of the strategic plan and with some final collection of data undertaken towards the end of the period, the data presented in this report is a summation up to December 2011. 171 Evaluation of the Reproductive Health Supplies Coalition Monitoring Framework Scope of the Monitoring Framework This Monitoring Framework is based on the Strategic Plan and targets indicators that are within the remit of Coalition partners to collect during the normal business of their work. Therefore the indicators selected were complied with the following criteria: Are already being collected by one or more Coalition partners (or agency of a partner) Are available for use by the Secretariat, free of data restrictions Are relevant to 14 focus countries receiving targeted support by the Reproductive Health Interchange (RHI), Project RMA (Resource Mobilization and Awareness), and the Global Programme of the United Nations Population Fund (UNFPA) Have a notional baseline value so that progress can be measured The matrix in Figure 1 was developed to illustrate simply the core logic and structure of the Coalition’s strategic plan. It was posted on the Coalition’s website in 2008 and has since become the point of reference used by most Coalition members, Working Groups and even the Secretariat to develop workplans, reference objectives and measures of performance (e.g. indicators), and rationalize the design and implementation of Coalition-inspired activities. This report pursues this trend and uses the matrix as the structural framework for this report. Figure 1. Schematic Overview of Coalition’s Strategic Plan (2007-2015). From the list of 58 illustrative indicators developed under the strategic plan—a refined list of 30 specific, operationalized and collectable indicators were developed to cover the focus areas detailed in Figure 1. Indicators were developed that had baseline values, a strategy for collection, and a means of verification. The full list of indicators is contained in Appendix A. Definition of country and global indicators The indicators in this Monitoring Framework draw on two distinct sets of data, each of which allows us to classify them as either “country” or “global” indicators. Country indicators are indicators whose values derive from data collected (or built-up) individually from 14 low to middle income (LMI) countries, each of which has served since the end of 2007 as the focus of at least one of three Coalition-sponsored initiatives: UNFPA Global Programme, Project RMA, and the RHInterchange. Unless indicated otherwise, use of the term “focus country” or “focus countries” in this framework refers specifically to this group of 14. Country-indicators, therefore, are designed to measure the impact of Coalition efforts on the ground—specifically where the Coalition, through the action of its members, can claim some direct attribution. 172 Evaluation of the Reproductive Health Supplies Coalition Table 1: Coalition Focus Countries Region Country RHI Project RMA Global Programme Africa Burkina Faso X X Ethiopia X X Ghana X X Mozambique X Rwanda X Tanzania X Uganda X Central and Latin America Guatemala X Honduras X Mexico X Nicaragua X X Asia Bangladesh X Mongolia X Nepal X Global indicators, by contrast, are indicators whose values derive from data aggregated at a global level. Although the content of these indicators may also speak to actions by countries or to changes occurring within them; they are especially useful for monitoring the evolution of global institutions or global phenomena (e.g. the actions of international donors, changes in international financial flows, etc.). The aim of global indicators is to portray supply-related changes within a universe, larger than the 14 focus countries alone. Collection of Indicators Many of the indicators are being routinely collected by various Coalition member organisations —some of which were transferred in their entirety and others that were extracted from larger data sets and refined for Coalition use. Indicators directly measuring the Coalition Secretariat’s performance were derived from the Secretariat itself. Throughout this report, unless otherwise stated, the results pertain to the three year period since the publication of the Coalition’s 2007-2009 Monitoring Report. Results Organisation of the results The Vision of the Reproductive Health Supplies Coalition is for all people in lower- and middle- income countries to be able to access and use affordable, high-quality supplies, including a broad choice of contraceptives, to ensure their better reproductive health. The results are thus presented under the theme of each goal and comprise a mixture of global and country level data, as well as quantitative and qualitative observations. Goal 1: Increase availability, predictability, and sustainability of financing for RH Supplies The Coalition and its members aim to increase resource flows for RH supplies from all sources at the global and country levels, including donors, consumers, public sector, and the private sector including the commercial sector. This goal of expanding the resource base for supplies is achieved by pursuing three strategies or “objectives”: 173 Evaluation of the Reproductive Health Supplies Coalition Objective 1.A: Increase the international flow of new resources to meet countries’ supply needs. The resources can come from public and non-governmental development assistance or from any one of a growing number of non-traditional funding sources (such as GFATM). Objective 1.B: Ensure that countries and their governments make use of the resources already at their disposal. This is achieved by encouraging national leaders to prioritize commodity security and translate that commitment into action. Objective 1.C: Encourage full use of the total market for RH supplies. Maximizing the total market increases overall total demand for RH services and supplies; grows the total resource base; facilitates sustainability; and ensures the most efficient use of public sector resources. Objective 1.A. Increase flow of new resources To gauge success in achieving Objective 1.A, increasing the international flow of new resources to meet countries’ supply needs, the Coalition’s Monitoring framework employs the following six indicators: 1A.1 Number of focus countries with a national budget line for RH supplies 1A.2 Percentage increase in donor funding for contraceptives in focus countries 1A.3 Percentage increase in donor funding for contraceptives worldwide 1A.4 Increase in number of donors making first-time contributions for RH supplies 1A.5 Number of GFATM country proposals per year with provisions for the purchase of RH supplies 1A.6 Increase in level of funding allocated to RH supplies by GFATM Indicator 1A.1 Number of focus countries with a national budget line for RH supplies Between 2007 and 2009, the date of the Coalition’s last Monitoring Report, the number of focus countries with national budget lines for RH supplies increased from 6 to 13. The only country without such a budget line was, and still remains, Mexico. It is not likely, however, that this situation will change insofar as commodity procurement in Mexico is de-centralized, with ultimate responsibility falling to the States, rather than the Federal government. For that reason, this indicator can be considered exhausted. Indicator 1A.2 Percentage increase in donor funding for contraceptives in focus countries Data from the Reproductive Health Interchange (RHI) reveal that donor funding for contraceptives in the Coalition’s 14 focus countries has increased by a total of almost 67 percent over the period 2006 to 2011. This represents a significant overall increase, but it also betrays significant variations from year to year, with funding levels rising dramatically in the first two years, but then levelling off in the remaining three. Indicator 1A.3 Percentage increase in donor funding for contraceptives worldwide Table 1: Increase/decrease in donor funding for contraceptives in focus countries Year Total Value Variation from previous year 2006 $47,994,710 N/A 2007 $70,387,981 46.7% 2008 $87,043,080 23.7% 2009 $81,336,476 -6.6% 2010 $70,529,678 -13.3% 2011 $80,157,728 13.7% Source: RH Interchange 174 Evaluation of the Reproductive Health Supplies Coalition Data from the Reproductive Health Interchange (RHI) reveal that donor funding for contraceptives worldwide increased by a total of almost 65.21 percent over the period 2006 to 2011—comparable to the increase seen in the 14 focus countries. Table 2: Increase/decrease in donor funding for contraceptives in focus countries Year Total value Variation from previous year 2006 $132,829,262 N/A 2007 $182,819,243 37.6% 2008 $200,454,303 9.6% 2009 $218,165,257 8.8% 2010 $202,636,720 -7.1% 2011 $219,168,173 8.2% Source: RH Interchange Indicator 1A.4 Increase in number of donors who make first-time contributions for RH supplies The monitoring tools currently in use by Coalition members offer only partial insight into the advent on new donor support for RH supplies. The RH Interchange, for example, does track funding sources, but the appearance of a new donor in the database does not necessarily signify the start of that donor’s support for commodity procurement. Many donors and procurers provide commodity support long before they begin reporting to the RH Interchange. UNFPA’s Donor Support for Contraceptives and Condoms also provides insight into global levels of support, but it does not serve as a longitudinal guide to the advent or particularly discontinuation of individual donor support. For that reason, the present indicator has been operationalized as the number of donors making first time contributions to UNFPA’s Global Programme for Reproductive Health Commodity Security (GPRHCS), which does track, on an annual basis, donor commitment. Since publication of the Coalition’s 2009 Monitoring Report, the GPRHCS has seen the number of ever-donors to the GPRHCS increase from seven to eleven, with the arrival of four new donors: Canada in 2009; France in 2010; and Australia and Denmark in 2011. Two of these new donors, Denmark and France, also joined the Coalition.14 Indicator 1A.5 Number of country proposals per year to the GFATM that make explicit provision for the purchase of RH supplies Tracking the use of Global Fund resources for contraceptive procurement is one way of measuring an expansion of the “non-traditional” resource-base for commodities. It is also a measure of the success with which Coalition members, during 2008 to 2010, assisted countries to incorporate RH commodities into their Global Fund proposals. It is, however, a challenging indicator. Between 2006 and 2010 (the last round of proposals to the Global Fund) approximately 36 HIV/AIDS program area proposals were approved per year. Some of these proposals made explicit provision for the procurement of RH supplies (defined here as both condoms—male and female—and other contraceptives), others did not. Either way, such provisions offered no _________________________ 14 UNFPA (2012). The Global Programme to Enhance Reproductive Health Commodity Security, Annual Report 2011. Page 41. 175 Evaluation of the Reproductive Health Supplies Coalition guarantee that resources would ever be spent on RH supplies; nor did their absence preclude supply expenditures at a later date. The most accurate and comprehensive way to track expenditures is through the Global Fund database. But while the Global Fund requires countries to report procurement of ARVs and condoms, it does not require the same of other RH supplies. Consequently, neither a content review of individual proposals, nor a review of the Global Fund database provide the information needed to track over time the use of Global Fund resources for RH supplies other than condoms. In this report, therefore, two sets of data are used to gauge the value of Global Fund resources spent per year on RH supplies. One set of figures, drawn from the Global Fund database, tracks the procurement of condoms per year. These figures do not include RH supplies other than condoms, nor do they necessarily reflect what has been distributed in a given year. The second set of data—and the only one that permits any disaggregation of non-condom RH supplies—comes from information contained in the RH Interchange and USAID|DELIVER Contraceptive Security Indicators (2011). The latter identifies countries reporting use of Global Fund resources for non-condom RH supplies. The RH Interchange, meanwhile, tracks commodities (condom and non-condom) financed by the Global Fund, but only commodities procured by agencies that report to the RHI. These procurements represent a subset of all Global Fund procurements and therefore the value of these is considerably lower. However, despite such variance, the overall trend lines from 2006 to 2011 remain comparable across both data sets. As Table 3 reveals, irrespective of the data source, the value of RH supplies (condoms only, condoms + other supplies) and the number of countries procuring RH supplies with Global Fund resources have increased steadily from 2006 to 2010. Indicator 1A.6 Increase in level of funding allocated to RH supplies by GFATM As shown in Table 3, the period 2006 to 2010 also saw dramatic increases in spending on RH supplies, and even on supplies other than condoms (such as implants, oral contraceptives, and injectables). The year 2011, the most recent year for which data is available, shows a decline in both spending and in the number of countries having spent Global Fund resources of RH supplies. The reasons for this drop may be attributable to the shrinking pool of Global Fund Table 3: Use of Global Fund resources to procure RH supplies 2006 2007 2008 2009 2010 2011 Value of condoms procured * 817,10 6 1,056,35 2 9,360,1 18 10,383,8 81 8,576,25 1 2,636,6 94 Value of RH supplies other than condoms * 84,247 894,144 0 Total value of RH supplies procured * 817,10 6 1,056,35 2 9,360,1 18 10,468,1 28 9,470,39 5 2,636,6 94 Countries procuring all RH supplies * 6 9 16 18 20 9 Value of condoms procured ** 200,47 2 2,154,46 5 7,181,8 97 9,822,87 1 17,359,6 05 8,019,5 25 Countries procuring condoms ** 1 3 27 52 53 31 Source: * RH Interchange and USAID|DELIVER Contraceptive Security Indicators (2011) ** GFATM Price and Quality Reporting’ (PQR) system: http://www.theglobalfund.org/en/procurement/pqr/ 176 Evaluation of the Reproductive Health Supplies Coalition resources, the absence of a new funding round in 2011, and perhaps more importantly, to the delays in reporting. The bulk of data among the 2011 Contraceptive Security Indicators is limited to 2009-2010. Objective 1.B. Access existing resources In 2009, research carried out under the auspices of the Coalition demonstrated the growing gap between the global demand for RH supplies and the availability of donor support to meet that demand. Under Objective 1A, the Coalition and its members seek to close that gap by increasing the resource base, particularly via traditional bi-lateral support and through other sources such as the Global Fund. But many countries already have at their disposal significant resources that could be channeled towards procurement of RH supplies. These resources may consist of locally generated revenues or even donor resources that effectively enter the national budget through budget and/or sector support. Under Objective 1.B, the Coalition and its members work to ensure that national governments commit to using local resources for RH supplies and that they follow through on those commitments. Two indicators are used to monitor this objective: 1B.1 Proportion of focus country budget line(s) for RH supplies actually spent 1B.2 Examples of new domestic financing mechanisms for supplies proposed or introduced in focus countries Indicator 1B.1 Proportion of focus country budget line(s) actually spent While nearly all focus countries today possess national budget lines for the procurement of contraceptive commodities, tracking the expenditure of that budget line has been difficult. Indeed, in the Coalition’s 2009 Monitoring report, only one country – Uganda – offered information on budget line expenditures. Fortunately, today, the data landscape is different. The importance of this indicator as a barometer of country commitment has prompted many Coalition partners to begin tracking it. USAID|DELIVER has introduced the variable into its annual Contraceptive Indicators tool. UNFPA, under its monitoring of the GPRHCS, has begun tracking budget commitments and expenditures for its Stream One countries. And a number of other agencies are now systematically tracking the indicator to inform advocacy efforts. These include the Advance Family Planning Project, funded by the Bill & Melinda Gates Foundation, as well as the Healthy Action Initiative, funded by the European Union (EU) and implemented by the German Foundation for World Population (DSW) and Institute for Education in Democracy (IED). The advent of these new data sources today offers us with far greater visibility into government expenditures on RHCS. Table 4, provides an overview of the data available for the 13 Coalition focus countries that possess a budget line item for supplies. Insofar as the results remain incomplete for both 2009 and 2011, it is hazardous at this stage to discern trends. But the few selected upward curves illustrated in Figure 2, are grounds for optimism that many countries are now meeting, and in some cases, dramatically exceeding their commitments. Of course, it also shows that some under-spenders, such as Mozambique, seem to be carrying on as before. Evaluation of the Reproductive Health Supplies Coalition Figure 2: Budget line allocations for RHCS in focus countries. 177 Evaluation of the Reproductive Health Supplies Coalition Figure 2: Budget line allocations for RHCS in focus countries. 178 Evaluation of the Reproductive Health Supplies Coalition Table 4: Government budget for contraceptives in 13 focus countries: Allocations and expenditures 2008 2009 2010 2011 Amount Allocated Amount Spent % Spen t Amount Allocated Amount Spent % Spen t Amount Allocated Amount Spent % Spen t Amount Allocated Amount Spent % Spen t Bangladesh* 714,285 36,428,57 0 5,100 Burkina Faso** 978,261 937,051 96 1,326,08 7 1,543,16 1 116 652,174 652,173 100 1,000,00 0 1,000,00 0 100 Ethiopia** 1,745,21 3 1,745,21 3 100 3,200,00 0 3,200,00 0 100 3,581,84 9 3,581,849 100 6,659,50 0 6,659,50 0 100 Ghana* 1,000,00 0 1,237,550 124 Guatemala* 1,500,00 0 1,500,000 100 Honduras* 1,270,17 1 2,699,112 212 Mongolia** 47,000 46,022 98 41,188 41,188 100 55,000 52,337 95 Mozambique* * 510,000 0 0 333,079 0 0 392,913 0 0 379,962 0 0 Nepal* 2,840,00 0 2,590,642 91 Nicaragua** 110,158 591,665 537 208,723 227,500 109 321,935 321,935 100 765,940 2,025,89 1 264 Rwanda* 600,000 1,454,420 242 Tanzania* 2,333,33 3 6,800,000 291 Uganda*** 848,837 54,651 6 872,093 0 689,655 0 0 2,480,47 8 3,307,30 4 133 TOTAL 148 70 112 115 * Data source: USAID|DELIVER Contraceptive Security Indicators (2011) ** Data source: UNFPA (2012). The Global Programme to Enhance Reproductive Health Commodity Security, Annual Report 2011. *** Data source: Moses Muwonge (2011). Tracking Government of Uganda Financing for RH Supplies. Partners in Population and Development/ARO 179 Evaluation of the Reproductive Health Supplies Coalition 1B.2 Examples of new domestic financing mechanisms for supplies proposed or introduced in focus countries In early 2011, the Secretariat commissioned, comprehensive reviews of the RH supplies environment in all of the Coalition’s 14 focus countries. The goal of this exercise was to report on key Coalition indicators in a newly developed set of “Country Profiles”, now located on the Coalition’s website. That review did not uncover any evidence of new domestic financing mechanisms being implemented or planned. It is not known whether any have been put into place since. Objective 1.C. Mobilize Total Market Resources From the outset, the Coalition and its members have encouraged full use of the total market for RH supplies. Maximizing the total market increases overall demand for RH services and supplies, grows the total resource base, facilitates sustainability and ensures the most efficient use of scarce public sector resources. Two indicators are used to monitor this objective: 1C.1. Examples of new initiatives undertaken in focus countries to expand the total market for RH supplies 1C.2. Increase in the non-public sector share of the market for condoms, pills, IUDs, and injectables in focus countries 1C.1 Examples of new initiatives undertaken in focus countries to expand the total market for RH supplies Between 2009 and 2011, the Coalition provided technical and financial support (under the Innovation Fund) to two large initiatives aimed at expanding the total market for RH supplies in Madagascar and Honduras. Implemented under the auspices of the Market Development Approaches Working Group, the two initiatives have informed similar work in Paraguay and the MDA WG continues to identify opportunities for scaling-up the work in under-resourced countries, including Chad, Mali, and Niger. Madagascar Total Market Initiative (Marie Stopes International, the Futures Group, and UNFPA), US$148,306 This initiative demonstrated the value to be derived from close collaboration among Coalition members at the country and global levels. The Futures Group’s technical expertise along with, Marie Stopes’ local knowledge of market segmentation, and the backing of UNFPA and the Ministry of Health, led to broad acceptance of the total market approach despite initial political uncertainty. The project demonstrated the adverse long-term effects of a government policy to provide free contraceptives, showing how this policy encouraged unnecessary reliance on public funds and services by clients with the means to pay. It also showed that the costs of transporting commodities to service-delivery points, once covered by user fees, were being borne by resource-poor districts, forcing them to compensate by cutting back on procurement. Through a strong participatory process, the study’s recommendations were adopted by the Ministry of Health, which then made the case before the country’s Council of Ministers. This political ownership forms the basis for continued engagement and the development of joint strategies and guidelines. Honduras Total Market Initiative (Abt Associates, John Snow, Inc.), US$188,000 This initiative forged new connections and built broad-based local partnerships across the public and private sectors. It also secured the Ministry of Health’s leadership of a total market approach, thereby overcoming an initial mistrust of the private sector. The comprehensive process drew together a diverse group of collaborators, including private laboratories, drug regulation authorities, national statisticians, the local society for obstetricians and gynecologists, and the country’s college of medicines. The project also led to the re- 180 Evaluation of the Reproductive Health Supplies Coalition establishment of the country’s Contraceptive Security Committee, which has since taken on the role of coordinating new efforts in contraceptive financing, procurement, and distribution. Indicator 1C.2 Increase in the non-public sector share of the market for condoms, pills, IUDs, and injectables in focus countries In the Coalition’s 2009 Monitoring Report, it was not possible to demonstrate any change in the non-public market share for selected contraceptives because none of the focus countries had completed a Demographic Health Survey (DHS) in the three year (2006 -2009) period since the launch of the Coalition. By 2011, four focus countries had completed a DHS and the results are reported below in Table 5. While Coalition efforts at market segmentation should ideally favor increases in the non-public share of the market for selected commodities; the current data set, limited as it is, does not reveal such a trend. Much of that may have to do with the countries themselves. Two the four— Ethiopia and Rwanda—have demonstrated remarkable increases in contraceptive prevalence over the past five years and the principle factor underlying that increase has been strong public sector support. It is not surprising therefore, that as a percentage of the all services delivered, the contribution of the private sector would have shrunk accordingly. By contrast, in Tanzania and especially Nepal, the share of the private sector has increased. Table 5: Percentage distribution of modern contraceptives by non-public sector share Country 2005-2006* 2010-2011* Ethiopia 20.5% (2005) 14.7% (2011) Rwanda 27.4% (2005) 6.6% (2010) Tanzania 31.1% (2005) 32.3% (2010) Nepal 21.3% (2006) 29.1% (2011) Source: DHS reports Goal 2. Strengthen capacity of health systems to deliver RH supplies in a sustainable manner Securing RH supplies relies on functional systems for forecasting needs, procuring, warehousing, distribution, and managing logistics information. Under the aegis of the Reproductive Health Supplies Coalition, members and their technical assistance partners are working with country counterparts to develop the tools and data needed to strengthen global and country RH supply chains, and ensure an efficient process for pre-qualification of RH products, manufacturers, and production facilities. Coalition members are also coordinating their efforts to best match country needs with the available technical assistance. Under Goal 2, the systems required to sustainably deliver supplies cover the spectrum of scenarios in the Global South: from relative stable political and economic contexts, to otherwise stable contexts experiencing period stock shortages, to countries suffering from conflict, natural disasters, and other crises. Across all three contexts, the Coalition and its members work to strengthen the capacity of health systems to deliver RH supplies in a sustainable manner. They do this by pursuing three strategies or “objectives”: 181 Evaluation of the Reproductive Health Supplies Coalition Objective 2.A: Ensure the integrity of systems for supply chain management, both globally and at country level. This is achieved by establishing new systems and/or by strengthening existing ones. Objective 2.B: Provide a coordinated, short-term response to stock-outs, supply disruptions and other failures that occasionally befall functional supply chains. Objective 2.C: Promote effective delivery of RH health supplies in settings where routine systems for supply chain management have broken down or ceased functioning. Objective 2A. Strengthen existing systems Of all the objectives within the Coalition’s Strategic Plan, strengthening existing systems maintains the largest number of indicators – eight altogether: 2A.1 Number of focus countries with functioning RH commodity committees or similar mechanisms for managing RH supply chains 2A.2 Number of focus countries that include RH commodities on the national essential drug list 2A.3 Examples of formal, coordinated actions by focus countries to improve RH supply chain performance (e.g. SPARHCS, national needs assessments, logistics systems strategies) 2A.4 Number of RH products from specific producers/factories, pre-qualified under WHO/UNFPA Programme 2A.5 Number of all countries (and focus countries), per year, with active registered users of the RH Interchange 2A.6 Decline in stock out rates at public-sector service delivery points in focus countries 2A.7 Descriptions of private sector supply chains for RH supplies in the private sector and use in the public sector 2A.8 Amount of funding provided through mechanisms that make RH supply resources less volatile Indicator 2A.1 Number of focus countries with functioning RH commodity committees or similar mechanisms for managing RH supply chains By 2007, seven of the Coalition’s fourteen focus countries already had functioning RH commodity security committees in place. By 2009, the remaining seven had either established or revived formerly dormant committees. As of 2011, all committees were still operational, thereby effectively exhausting this indicator. Indicator 2A.2 Number of focus countries that include RH commodities on the national essential drug list Another country-level system affecting the availability of RH supplies is the inclusion of contraceptives on the countries’ essential drugs list (EDL). By 2009, all focus countries included contraceptives on their EDL. However, it is worth noting that in the two year period since the Coalition’s last Monitoring Report, the types of commodities included in the lists have doubled (see Table 6). Table 6: Number of focus countries with RH commodities on the country’s essential drug list Baseline Jan 07 2009 2011 Number of types of contraceptives on EDL* 2009 Number of types of contraceptives on EDL* 2011 Burkina Faso No Yes Yes 7 8 Ethiopia Yes Yes Yes 7 9 Ghana No Yes Yes 1 8 Mozambique No Yes Yes 6 7 Rwanda Yes Yes Yes 8 9 182 Evaluation of the Reproductive Health Supplies Coalition Indicator 2A.3 Examples of formal, coordinated actions by focus countries to improve RH supply chain performance (e.g., SPARHCS, national needs assessments, logistics systems strategies) As shown below in Table 7, eleven focus countries have undertaken formal coordinated efforts to improve RH supply chain performance—the bulk of these efforts having taken place from 2009 to 2011. Nearly all the efforts have involved either SPARHCS analyses (5) or the formulation of national RHCS strategies. Copies of all these documents are available on the Coalition’s website, either through SID (Supply Information Database) or the Country Pages. Indicator 2A.4 Number of RH products prequalified under WHO/UNFPA Programme A number of global level systems have a direct bearing on the availability of RH supplies in country and one of the most important is the Prequalification of Essential Medicines undertaken by WHO—and linked to this the UNFPA-led scheme for prequalification of medical devices for RH (condoms and IUDs). The rationale here is that donors, governments, social marketing organisations—anyone who is procuring RH supplies—need to be confident that they are procuring commodities that have an assured quality. In 2006, members of the Coalition agreed to procure only prequalified commodities, where there are at least two pre-qualified suppliers (Bonn meeting, 2007). Tanzania No Yes Yes 2 7 Uganda No Yes Yes 4 6 Guatemala No Yes Yes 2 6 Honduras Yes Yes Yes 5 5 Mexico Yes Yes Yes 1 N/A+ Nicaragua Yes Yes Yes 6 4 Bangladesh No Yes Yes 2 7 Mongolia+ Not known Yes Yes N/A N/A Nepal Yes Yes Yes 4 8 Total (if app) 5 13 13 42 84 + Detailed information not available. * Contraceptive types include: Male Condoms, Female Condoms, Combined Pill, Progestin-only Pill, Emergency Contraception, Injectable, Implant, IUD – of any specification Source: CS Indicators from JSI/DELIVER Table 7: Formal, coordinated actions by focus countries to address RHCS Country SPARHCS RHCS Strategy Other Bangladesh 2006 Burkina Faso 2009 Ethiopia 2010 2006 2011 Ghana 2011 Guatemala 2010 Mozambique 2011 Nepal 2006 2007 Nicaragua 2009 Rwanda 2006 2008 Tanzania 2007 Uganda 2008 2010 183 Evaluation of the Reproductive Health Supplies Coalition While much work remains to be done in increasing the number of prequalified products, progress is being made. As shown below in Table 8, the number of prequalified condoms has more than doubled since 2007; the number of IUDs has quadrupled; and there are now eleven different hormonal contraceptives pre-qualified, including three from generic manufacturers. Table 8: Number of RH products pre-qualified by WHO/UNFPA 2007 2010 Male Condoms 10 26 Female Condoms+ 2 1 IUDs 2 8 Hormonals 0 11 Source: http://apps.who.int/prequal/ Indicator 2A.5 Number of all countries (and focus countries) per year, with active registered users of the RH Interchange Originally developed under the auspices of the Systems Strengthening Working Group, the Reproductive Health Interchange (RHI) has evolved into an extremely valuable database, tracking shipments of supplies to over 140 countries. Though most traffic to the RHI is referred from the Coalition website, the database has been, since 2011, housed at UNFPA. It forms an integral part of the new procurement mechanism, AccessRH. As shown in Table 9, the last four years have seen the RHI grow tremendously, both in terms of countries with registered users and the absolute number of users. All of the Coalition’s 14 focus countries have registered users of the tool. Indicator 2A.6 Decline in stock-out rates at public-sector service delivery points in focus countries Effective supply-chains should, in principle, contribute to reductions in the level of stock-outs at both central- and service-delivery levels. For that reason, the Coalition’s Monitoring framework sees the decline in stock-out rates at public-sector service delivery points (SDPs) as a key indicator of supply chain performance. Once again, a number of challenges exist in securing the data required to monitor this indicator. First of all, SDP stock-out data are available from only 9 of the Coalition’s 14 focus countries. Secondly, the two key agencies measuring and reporting this data—USAID and UNFPA— define and measure stock-outs differently. UNFPA, for example, tracks the percentage of SDPs Table 9: Registered users of RH Interchange 2007 2009 2011 Number of focus countries with registered users 2 12 14 Number of countries with registered users 7 65 92 Number of registered users 29 304 632 Source: Jane Feinberg, AccessRH, personal communication 184 Evaluation of the Reproductive Health Supplies Coalition reporting “no stock-outs of contraceptives within the last six months,” USAID tracks the frequency of stock-outs by method (male condom, COC, injectables).15 It also does so quarterly—while UNFPA tracks every six months. Table 10, therefore, consolidates the figures of both agencies, while framing the variable in the same manner as UNFPA: percentage absence of stock-outs. Nonetheless, because of differences in what is being measured, country data from UNFPA are reported on a single line, while those from USAID are disaggregated by method: male condom, combined oral contraceptives (COC), and injectables.16 _________________________ 15 In Table 10, quarterly data from USAID have been averaged to provide a single annual figure. 16 Despite the limitations described in the narrative, UNFPA and USAID are the only agencies that systematically track stock out data on most Coalition focus countries – and for that reason, the Coalition relies on their data to monitor this strategic indicator. Nevertheless, these figures do stand in sharp contrast to high stock-out levels being reported in one-off surveys undertaken by the Bill & Melinda Gates Foundation. Again, the use of different methodologies, definitions, and catchment areas among all three groups illustrate the complexity in comparing different data sources. 185 Evaluation of the Reproductive Health Supplies Coalition Indicator 2A.7 Descriptions of private sector supply chains for RH supplies in the private sector and use in the public sector Coalition efforts at strengthening health systems have, to date, focused overwhelmingly on the public sector—on the development of tools to strengthen information systems, overcome inadequate financing, and redress limited human resources. And yet in many countries, these weak public supply chains stand alongside a thriving and efficient private sector—where stock- outs are the exception; where financing is adequate; and where information flows are smooth and efficient. Learning what makes private sector supply chains work and maximizing opportunities for increased public-private cooperation are two strategies for strengthening health systems. This indicator measures the degree to which they are being used by Coalition members. Table 10: Percentage of SDPS in nine focus countries reporting 'no stock-outs' of contraceptives 2007 2008 2009 2010 2011 Bangladesh* Male condom 100 50.5 100 COC 47 100 100 Injectables 100 99 97.9 Burkina Faso** All methods 29.2 81.3 12.8 Ethiopia* All methods 60 90 99.2 98.8 Honduras* Male condom 75.4 COC 72.1 Injectables 76 Mongolia** All methods 100 100 97.6 37.7 Mozambique** All methods 24.1 81 Nepal* Male condom 90 93.6 94.7 96 97.9 COC 94 96.3 96.7 97.5 97.5 Injectables 96.5 97.9 97.7 98.5 99 Nicaragua** All methods 66 81 99.7 64.5 Rwanda* Male condom 98 98 95.3 96.8 99 COC 97.2 96.6 95 95.1 98.9 Injectables 98 98 95.3 96.8 99 * Source: USAID | DELIVER ** Source: UNFPA 186 Evaluation of the Reproductive Health Supplies Coalition In December 2010, Bayer HealthCare and USAID launched a joint "Contraceptive Security Initiative" with the goal of providing middle-income earners across sub-Saharan Africa with increased access to Micrgynon® Fe, the most widely used oral contraceptive in the world. The plan is to create an affordable market for Microgynon® Fe based on local prices—but prices that provide Bayer HealthCare with enough of a margin to sustain the program, without third-party involvement. The program was launched in Ethiopia, replicated in Uganda and Tanzania, and soon to be rolled out in eight other sub-Saharan nations. Indicator 2A.8 Amount of funding provided through mechanisms that make RH supply resources less volatile. In 2010 and 2011, the Coalition saw the launch of two new financing and procurement mechanisms aimed at reducing the volatility of RH supply resources. AccessRH: In 2010, UNFPA’s Procurement Services Branch launched AccessRH, an innovative procurement mechanism that reduces volatility in supply resources though high-volume procurement, warehousing of commodities for rapid delivery, and transparent information on pricing and product costs. The product of collaborative design and financing by Coalition members, AccessRH secured support from multiple donors in 2010, built its staff, and established a core revolving stock fund. As shown in Table 11, the value of orders placed through AccessRH has to date exceeded US$32 million including Third Party and UNFPA Country office procurement. Pledge Guarantee for Health: In early 2011, the UN Foundation launched the Pledge Guarantee for Health (PGH), an innovative financing mechanism designed to help recipients turn unrealized aid commitments into short-term, low-cost commercial credit for the purchase of reproductive health commodities. In early 2011, the PGH enabled the Zambia Ministry of Health to procure US$4.8 million of insecticide-treated bed nets in advance of the rainy season, and in so doing, avert an estimated 100,000 cases of malaria. In late 2011, PGH mobilized US$12 million for the government of Ethiopia―an amount that allowed it, in early 2012, to place the largest single order of implants (600,000 units) in the history of US pharmaceutical manufacturer Merck. Objective 2B. To avert RH supply disruptions at national level—help when systems break down Indicator 2B.1 Description of stock-outs averted or resolved through the intervention of the Coordinated Assistance for RH Supplies (CARhs) Group Table 11: Order profile of Access RH since inception Procurement Type 2011 2012 (Q1) Jan 1 2011- Mar 31 2012 US$ Qty US$ Qty US$ Qty Third Party 21,942,806 66 5,589,160 17 27,531,966 83 UNFPA Country Office 3,730,838 52 796,803 8 4,527,641 60 Total 25,673,644 118 6,385,963 25 32,059,607 143 Source: Campbell Bright, UNFPA/PSB, Copenhagen 187 Evaluation of the Reproductive Health Supplies Coalition No activity better captures the Coalition’s ability to add value to the work of its members than the CARhs (Coordinated Assistance for RH Supplies) Group. The CARhs brings together the world’s key commodity suppliers to address the short-term supply crises that periodically befall countries. Through electronic data-sharing and decision-making, members identify existing or potential supply shortages or overstocks, assess their causes, develop solutions, and where possible, apply them. Between October 2009 and September 2010, the CARhs addressed 184 separate supply crises— the highest number of cases since the group was established in 2005. In 73 cases, members identified critical information that either shed light on the problem at hand or obviated the need for further action. In 65 percent of the 40 instances in which stock levels had dropped below minimal requirements, the CARhs successfully averted a full stock-out, either by issuing or expediting new shipments, or by providing policy advice. In 2010, such remedial efforts drove the procurement of more than $8.7 million in reproductive health commodities. In 2011, the CARhs Group addressed 168 separate supply crises, successfully resolving 61 percent of all cases requiring direct intervention. It saved money by arranging the transfer of more than US$2 million in excess commodities from Rwanda to Burkina Faso. It played a pivotal role in clearing millions of dollars’ worth of RH supplies stuck in port in Nigeria. In addition, it assisted with the procurement of six million condoms for a new military HIV/AIDS prevention program in Burundi. In November, the CARhs Group was awarded £10 million in support from DFID to allow it to address commodity shortfalls not otherwise amenable to existing financing mechanisms, and/or to avert emergency financing gaps before they occur. Objective 2C. Systems for emergency situations Indicator 2C.1 Number of Emergency RH kits sent to different countries by humanitarian agencies per year Part of the remit of the Coalition is to support, improve or design systems that provide RH supplies in emergency situations—be these natural disasters or in conflict settings involving internally displaced people and refugee populations. In this regard UNFPA has designed 18 different emergency kits17, which bundle together the RH supplies needed for everything from emergency obstetric care to a “PEP” kit (Post exposure prophylaxis). Between 2007 and 2008, 57 countries requested kits mostly from UNFPA Country Offices. Since then, the number has increased to 72. UNFPA Country Offices remain the largest single procurers of emergency kits, but other major procurers include Save the Children, Marie Stopes International, CARE, Merlin and IPPF, as well as other UN agencies (UNICEF, UNHCR, WHO). Table 12 indicates the total volume of RH emergency kits dispatched per year since 2007. Table 12: Volume of RH Emergency Kits dispatched (2009-2011), by region Regions Volume (pieces) 2007 2008 2009 2010 2011 Africa 3,242 4,214 4,283 Arab States 3,074 1,955 2,030 Asia and Pacific 802 4,830 1,775 LAC 563 1,660 215 _________________________ 17 Internally, UNFPA classifies the kits as EMR (emergency) and NEMR (non emergency), depending on the urgency with which they must delivered. The kits are, however, identical and are treated as such in Table 12. 188 Evaluation of the Reproductive Health Supplies Coalition HQ 0 0 122 E. Europe & C. Asia 49 295 109 Other 0 1 0 Total 8,990 9,165 7,730 12,955 8,534 Source: Daniela Andries, UNFPA/PSB, Copenhagen (Figures for 2007-8 from RHSC Monitoring Report (2009) Indicator 2C.2 Examples of Coalition-inspired initiatives to integrate RH supplies into humanitarian relief efforts Between 2009 and 2011, the Coalition provided financial assistance to Marie Stopes International (MSI) to examine commodity security at the service-delivery level in the Democratic Republic of the Congo, South Sudan, and Uganda. Implemented under the auspices of the Systems Strengthening Working Group, the study resulted in a better understanding of the challenges facing agencies that deliver emergency reproductive health supplies. It prompted MSI to install logistics coordinators at two sites and add logistics management to the job description of a program manager at a third site. The study also led them into a new partnership with the International Dispensary Association which aims to include comprehensive commodities on the standard International Dispensary Association list of emergency essential medical supplies. Finally, at the global level, the findings of the study were incorporated into the strategic planning efforts of the Reproductive Health Access, Information, and Services in Emergencies (RAISE) initiative, a broad partnership of humanitarian and development agencies, United Nations agencies, advocacy agencies, and academic institutions. Goal 3. Assure the added value of the Coalition as a productive and sustainable global partnership through efficiency, advocacy, and innovation This goal seeks to reinforce the capacity of the Coalition to reach out and bring together diverse stakeholders to share experience and expertise and catalyze joint initiatives, increase effective use of limited resources, serve as the technical resource and global advocate for RH supplies security, and generate and promote innovation—all essential to meeting the Coalition’s first two strategic goals. The Coalition and its members work to assure the Coalition’s added value by pursuing the following three strategies or objectives: Objective 3A. Foster greater harmonization among stakeholders at global and country level to generate economies of scale, maximize impacts, and minimize the risks of redundancy and confusion. Objective 3B. Generate, disseminate, and facilitate the application of knowledge through innovation, research, and documentation. Objective 3C. Offer a credible and sustainable institutional framework within which stakeholders can collaborate, build commitment to supply security, formulate strategies for change, develop common indicators of success, and measure progress against them. Objective 3A. Foster greater harmonization among stakeholders at global and country level To gauge success in achieving Objective 3A, the Coalition’s Monitoring framework employs the following three qualitative indicators: 189 Evaluation of the Reproductive Health Supplies Coalition 3A.1 Examples of coordinated action by any two or more Coalition members, in the same focus country, with the explicit aim of strengthening RH commodity security 3A.2 Examples of coordinated action by two or more Coalition institutions with the explicit aim of strengthening RH commodity security beyond the boundaries of a single country 3A.3 Examples of new Coalition tools and/or publications aimed at comparing or harmonizing comparable activities being implemented by two or more Coalition members. Indicator 3A.1 Examples of coordinated action by any two or more Coalition members, in the same focus country, with the explicit aim of strengthening RH commodity security Since 2009, at least 6 of the Coalition’s 14 focus countries have benefitted from coordinated action by two or more Coalition partners aimed at strengthening RH commodity security. Ethiopia: In 2011, the UN Foundation, Merck, World Bank, DFID, and the Ethiopia Ministry of Health worked together to enable the Pledge Guarantee for Health to mobilize US$12 million; thereby enabling the Ethiopian government to secure 600,000 units of the contraceptive implant, Implanon®. Honduras: A major catalyst of coordination among Coalition members has been the Innovation Fund, which has provided financing for a number of joint program efforts. One of these efforts is the Honduras Total Market Initiative, which saw Abt Associates and JSI join forces to build broad-based local partnerships among a diverse group of collaborators, including private laboratories, drug regulation authorities, national statisticians and others (see previous section for description of activity). Mexico: Four Coalition members (INSAD, UNFPA/Mexico, and the Mexican Secretariat of Health) are engaged in a joint research effort to document the impact of investments in contraception and family planning. The results of this work are designed to shape advocacy efforts of the LAC Forum’s own RMA Working Group. Nicaragua: In 2011, four members of the LAC Forum’s (see below) Professionalization Working Group (PRISMA, USAID|DELIVER, CIES-UNAN, and UNFPA/Nicaragua jointly conducted a needs assessment on Human Resource Capacity in Public Health Supply Chain Management. The results of this joint research initiative fed directly into the establishment of the People that Deliver Initiative, of which Nicaragua is a focus country. Tanzania: USAID, DFID, and AusAID have pooled resources to support the Tanzania Ministry of Health and Social Welfare’s family planning and maternal health programs. This joint effort aims, among other things, to reduce contraceptive shortages through coordinated purchasing and forecasting of family planning commodities, ensuring in a continuous pipeline of supplies through 2015. Uganda: While Uganda had been, from the outset, a major beneficiary of advocacy efforts under Project RMA (2006-2009), its selection as the venue for the Coalition’s 2010 Annual Meeting quickly attracted the attention of an even wider circle of Coalition members. These included the Advance Family Planning Project, Partners in Population and Development, and other Ugandan members of the East African RH Supplies Advocacy Network. Joint funding and program efforts among these members have lead to a number of recent successes, not the least of which was the 2011 decision of the Government of Uganda to increase by nearly four- fold its national budget line for RH supplies and—also a first—exceed expenditures on it by over US$1 million (see Table 4). 190 Evaluation of the Reproductive Health Supplies Coalition Indicator 3A.2 Examples of coordinated action by two or more Coalition institutions with the explicit aim of strengthening RH commodity security beyond the boundaries of a single country All of the Coalition’s global-level tools and initiatives, in one way or another, seek to effect change beyond the boundaries of a single country. The Coalition’s 2009 Monitoring Report, for example, singled out for attention the RH Interchange, which is sustained by the input of nearly 15 Coalition members and accessed by users from over 140 countries. It also discussed the work of the CARhs, which brings together a core of five to ten key donors and procurers with the aim of identifying and resolving reproductive health supply shortages at country-level. Between 2010 and 2011, the CARhs addressed to stock shortages in no fewer than 12 countries.18 A more recent example of Coalition-inspired cross-boundary efforts is the “People that Deliver” Initiative (PtD), a new initiative born out of the Systems Strengthening Working Group that seeks to strengthen the professional skills of supply chain mangers globally. Though its work is global, the initiative itself places emphasis on strengthening supply chain managers in a select group of about seven focus countries (Liberia, Burkina, Mozambique, Namibia, Ethiopia, Indonesia, Dominican Republic). Few activities, however, better illustrate the potential of the Coalition to strengthen RHCS across borders than the two new regional Forums: the Spanish-speaking LAC Forum (Latin America and Caribbean), and, more recently, the Francophone Forum, SECONAF. In 2010, the Secretariat took the lead in reaching out to potential Latin American and Caribbean partners through the formation of the LAC Forum. With its 59 member institutions representing 20 counties, the LAC Forum and its recently established LAC Fund (analogous to the Coalition’s Innovation Fund) provides a regional platform for discussions on commodity security among Coalition members and other stakeholders throughout the region. For example, a recent LAC Fund award to the Peruvian NGO Prisma, is enabling five Coalition members (Prisma, PASMO, ProSalud Bolivia, and USAID|DELIVER) to build the capacity of partners to identify barriers to RHCS in six countries. Indicator 3A.3 Examples of new Coalition tools and/or publication aimed at comparing or harmonizing comparable activities being implemented by two or more Coalition members In 2010, the Coalition launched the HANDtoHAND Campaign, an ambitious effort designed to maximize its institutional outreach and rally the support of the family planning community behind United Nations Secretary-General Ban Ki-moon’s Global Strategy for Women’s and Children’s Health. The HANDtoHAND Campaign comprised two core components. The first was the formulation of an ambitious but achievable metric to reduce unmet need for family planning by 100 million new users of modern contraception by 2015. The second was getting the family planning community’s buy-in on the new metric and securing the financial, programme, and technical commitments needed to achieve it. As a result of widespread dissemination, media attention and the publication of the analysis underlying the metric, the HANDtoHAND Campaign has proven to be a catalyst for joint commitment and coordinated action. By the end of 2011, the Campaign had secured more than 25 pledges of financial, programmatic, and policy support from across the public and private _________________________ 18 UNFPA (2012). Coordinated Assistance for Reproductive Health Supplies Group Progress Report: October 2010 – September 2011 (unpublished). 191 Evaluation of the Reproductive Health Supplies Coalition sectors. Furthermore, its 100 million metric was taken up as the cornerstone of the new five- year public-private global Alliance for Reproductive, Maternal, and Newborn Health (the Alliance) among the US Agency for International Development (USAID), UK Department for International Development (DFID), Australian Agency for International Development, and the Bill & Melinda Gates Foundation. Objective 3B. Generate, disseminate, and facilitate the application of knowledge through innovation, research, and documentation. Indicator 3B.1 Examples of all new tools, publications, and research initiatives produced each year, either by the Coalition or by its members in response to Coalition-inspired initiatives Every year, the Coalition and its members produce a range of tools, publications, presentations – all linked to activities and initiatives carried out under the auspices of the Coalition. Some of these, such as videos and presentations, are designed for advocacy purposes; some are designed to heighten global awareness of the Coalition as an institution; and some are designed to communicate the results of more technical initiatives. But equally as diverse as their content, is their format. Some materials are produced in hard copy, some electronically; some, such as SupplyInsider are produced monthly, while others are produced on a case by case basis. While most materials still tend to be produced in English, more and more are appearing in French and Spanish. In 2007, the Coalition launched an online reference library with the aim of providing access to documents on the status of reproductive health supplies at country-level. Beginning in 2011, that library was expanded to include the full range of materials produced by the Coalition and its partners. That work is still in process, many items are still awaiting entry, but many can already be accessed via the following link: http://www.rhsupplies.org/resources-tools/supplies- information-database-sid.html. A summary of key publications for the period 2009-2011 is provided in Table 13. Just as SID offers convenient access to the full-range of Coalition inspired publications, the Coalition’s tools can also be accessed online via the “Resources and Tools” page of the Coalition website. The following three tools have been launched since 2009: Country Profiles: In 2010, the Coalition launched a set of “country pages”, each designed to provide stakeholders within the RH community up-to-date information, both qualitative and quantitative, on reproductive health commodity security in the Coalition’s 14 focus countries. LAPTOP: In early 2012, the Coalition launched a new database of supply chain training courses. With courses on topics ranging from quantification to warehousing, essential medicines to transportation, LAPTOP seeks to enhance the profile of supply chain managers as critical players in the effective delivery of health commodities. It is today, the most visited page on the Coalition website. Innovation Fund: Since its launch in 2009, the Innovation Fund has been a major catalyst of innovation, research and documentation. Made possible by a generous grant from the Bill & Melinda Gates Foundation, the Innovation Fund has made available over US$2 million to member organizations in each of the Coalition’s three Working Groups. Table 14 provides an overview of the activities carried out under the Innovation Fund, while more detailed descriptions can be found on the Innovation Fund webpage: http://www.rhsupplies.org/resources-tools/innovation-fund.html 192 Evaluation of the Reproductive Health Supplies Coalition Table 13: RHSC Publications 1 Market Segmentation Primer (MDAWG) (2009) 2 Optimizing the Reproductive Health Supply Chain -- A User's Guide to Software (2009) 3 Contraceptive Projections and the Donor Gap- Meeting the Challenge (2009) 4 Optimizing the Reproductive Health Supply Chain: A User’s Guide to Software (2009) 5 Access for All -- The Secretariat of the RHSC Annual Report 2010 (2010) 6 Developing a Family Planning Goal for 2015 that Supports MDG-5b (USAID, 2010) 7 Empty Handed: Responding to the demand for contraceptives (Film, 2010, Population Action International) 8 Planned Parenthood Association of Ghana (Film, 2010) 9 TMI Madagascar Info Sheet (MDAWG) (2010) 10 Total Market Initiative - Honduras Stakeholders Meeting (Film, 2010) 11 UNFPA documents on Global Programme: Success Stories and RHCS Update (2010) 12 World Bank RH Action plan (2010-2015) 13 Access for All Call to Action (2011) 14 Access for All- Supplying a New Decade for Reproductive Health: A Report on the 2011 conference in Addis Ababa, Ethiopia (2011) 15 Caucus Product Briefs on various modern contraception methods, safe abortion methods, and HPV vaccine (Ten briefs originally published in 2009, updated 2011) 16 Reproductive Health Commodity Security: Leading from behind to forge a global movement (2011) 17 The Road from Istanbul to Addis and Beyond: Setting an Agenda for Reproductive Health Supplies (2011) 18 Sparking change through the Innovation Fund: Successes and lessons learned for reproductive health supplies (2011) 19 Skibiak J. For More Than 200 Million Women, A World Without Options. RH Reality Check. June 2, 2011. 20 Meeting the Challenge of a New Decade, a multimedia presentation on the past, present, and future of RH commodity security, developed in cooperation with the Population Reference Bureau and USAID. 21 Key data and findings on contraceptive and maternal health commodities for UN Commission (2012) 22 SupplyInsider (12 issues, in Spanish and English) 193 Evaluation of the Reproductive Health Supplies Coalition Table 14: Innovation Fund - Small Grants Program Organization WG Project Amount Start Date End Date Round 1 (Jan 2009) JSI SSWG Professional Development 67,007 3/1/2009 12/31/2009 MSI SSWG Emergency Settings 65,636 4/1/2009 12/1/2009 Abt Associates MDA TMI - Honduras 188,218 4/1/2009 5/31/2010 Round 2 (May 2009) E&P RMA Engage French-speaking Africa in RHS Advocacy 140,000 8/1/2009 9/30/2011 ACPD RMA RH Advocacy at Canada G8 71,760 8/1/2009 9/30/2011 PAI RMA RH Supplies Documentary 173,219 9/1/2009 11/30/2010 MSI MDA TMI - Madagascar 148,306 8/1/2009 12/31/2010 Round 3 (Sept. 2009) JSI SSWG Manufacturer Input to AccessRH 22,660 12/1/2009 3/31/2010 JSI SSWG Best Practices for Supply Chain Management 58,403 12/1/2009 10/31/2010 Concept Foundation MDA Accessing Quality Assured RH Supplies 198,043 12/1/2009 2/28/2011 Round 4 (Feb 2010) JSI SSWG Professional Development Opportunities II 121,979 6/1/2010 5/15/2012 Round 5 (June 2010) PAI RMA Istanbul +10 133,912 10/1/2010 12/31/2011 Round 6 (Oct 2010) BioForce SSWG Professionalization of RH supply chain mgmt 140,956 2/15/2011 6/30/2012 IPPF - London SSWG Strengthen capacity of systems to deliver RH 79,318 3/1/2011 9/30/2012 IRH - Georgetown SSWG Forecasting 161,998 3/1/2011 10/31/2012 Round 7 (Feb 2011) PRISMA Caucus Health Commodities 157,630 7/1/2011 3/31/2012 Round 8 (June 2011) None awarded 0 Total Funded to Date 1,929,044 194 Evaluation of the Reproductive Health Supplies Coalition Indicator 3B.2 Annual increase in visits to the Coalition’s website Traffic to the Coalition’s website has grown dramatically since its launch in 2008. By 2009, the date of the last Monitoring Report, website visits per month had jumped from an average of 547 in 2008 to approximately 1,600. In June 2011, the Coalition Secretariat introduced “pickwik”—a new web statistics software that excludes the traffic of search machine crawlers and therefore more accurately represents user hits. Because the analytics of pickwik are different from those used previously, comparing results across the two is unreliable. Nevertheless, by March 2012, website traffic had reached 2,437 hits per month—equivalent to a growth of 62.9% in nine months. The average number of hits per month since the introduction of pickwik is 1,738. The most visited sections of the website are the Working Group and the Resources & Tools sections. The most visited page, as of March 2012, was the portal to the newly launched LAPTOP, the database of supply-chain related professional development opportunities for public-sector health commodity managers. The Coalition website is also the key conduit to the website of AccessRH and nearly two-thirds of all traffic to the RH Interchange originates from the Coalition website. Social Media: In May 2011, on the eve of the Access for All meeting, the Coalition began using Twitter on a regular basis, both retweeting and sharing original content. Since then the RHSC account has secured 280 “followers” from the wider reproductive health, family planning, maternal health, global health and development community. The Coalition Secretariat “tweets”, on average, ten times per week. Objective 3C. Offer a credible and sustainable institutional framework The Coalition’s success in framing global efforts to achieve supply security are measured by five indicators, all of which reflect the Coalition’s role as an agency for adding value to the work of members, and members’ assessment of the success with which that role is performed. 3C.1 Completion of Monitoring reports in 2009 and 2015 3C.2 Number of institutions that contribute financially, each year, to the implementation of Coalition-sponsored or -inspired activities (excluding core support to Secretariat operations) 3C.3 Total financial contributions of all institutions, per year, to the implementation of Coalition-sponsored or -inspired activities 3C.4 Total financial contribution of all institutions, per year, to sustaining Secretariat’s core operating costs 3C.5 Total number of Coalition members and increase Indicator 3C.1 Completion of monitoring reports in 2009 and 2015, measuring core Coalition performance indicators against baseline values This report fulfils this indicator. Indicator 3C.2 Number of institutions that contribute financially, each year, to the implementation of Coalition-sponsored or inspired activities (excluding core support to Secretariat operations) Since 2007, the Coalition has leveraged millions of dollars in support of both its own work and the cause of RH commodity security more broadly. These leveraged resources are independent of the Secretariat’s own core operating costs. They are new resources, leveraged either in support of—or as a consequence of—initiatives undertaken under the umbrella of the Coalition. As shown below in Table 15, the number of contributing institutions has fluctuated significantly over the past five years. It peaked in 2007, then again in 2010, largely as a result of support to 195 Evaluation of the Reproductive Health Supplies Coalition the launch of the People that Deliver Initiative—once a workstream under the Coalition’s Systems Strengthening Working Group. The year 2011 saw the fewest number of contributors, but also yielded the highest volume of financial support. Table 15: Number of institutions contributing non-core support Year 2007 2008 2009 2010 2011 Number of Institutions 21 14 12 20 7 The financial contributions of the each partner organization are detailed below in Figure 15. Indicator 3C.3 Total financial contribution of all institutions, per year, to the implementation of Coalition-sponsored or inspired activities (excluding core support to Secretariat operations) As noted previously, the Coalition has leveraged millions of dollars in support of both its own work and the cause of RH commodity security more broadly (Table 16). In 2011, the Coalition leverage more US$25 million in financial support, more than doubling the previous year’s figure of US$10.6 million which, until then, had been the highest since the Coalition’s establishment in 2004 (see Figure 3). The year 2011 also marked a turning point in the way such resources are leveraged. Until 2011, most additional resources had been leveraged as supplementary support to Coalition-inspired activities: additional funding to meetings, donor contributions to new Coalition work streams, or the scale-up of initiatives launched with Coalition-provided seed money, for example. In 2011, for the first time ever, Coalition initiatives themselves yield nearly US$2 million in added value, specifically from the savings generated by decreases in the prices of contraceptive implants sold in 2011. Just weeks into 2012, these costs savings were further augmented as Bayer again reduced the price of its contraceptive implant, Jadelle®, and Merck/MSD announced higher than expected sales of its implant, Implanon®. The savings attributed to these developments, combined with those of 2011 yielded a total of $9.3 million in cost savings. Figure 3. Stakeholder contributions and new resources mobilized. 1,286,401 5,304,085 7,923,000 10,665,000 23,523,243 1,822,362 0 5,000,000 10,000,000 15,000,000 20,000,000 25,000,000 2007 2008 2009 2010 2011 Stakeholder contributions New resources mobilized 196 Evaluation of the Reproductive Health Supplies Coalition Table 16: Total funds leveraged by activity and institution per year 2009 AccessRH Bill & Melinda Gates Foundation $300,000 UNFPA $300,000 USAID (JSI Secondment) $250,000 Pledge Guarantee for Health Bill & Melinda Gates Foundation $6,800,000 Countries at Risk Group (SSWG) USAID, UNFPA, RHI, MSI, KfW, World Bank—staff time $80,000 Market Segmentation Primer (MDA WG) Abt Associates and others $30,000 May 2009 Membership Meeting (London) IPPF (Participant travel and MDA WG costs) $35,000 MSI—MDA WG and Caucus meeting costs $5,000 Coalition Web-site and Information Developments USAID (through DELIVER)—Country Pages $40,000 USAID (through DELIVER)—"Istanbul+10" $80,000 Secretariat Staff Travel IPPF (Kinzett to Dacca; June 2009) $2,500 Universal Access to Female Condoms (Skibiak to Amsterdam; Oct 09) $500 TOTAL $7,923,000 2010 AccessRH UNFPA, USAID $10,250,000 CARhs USAID, UNFPA, RHI, MSI, KfW, World Bank $84,000 Market Development Approaches WG quality initiative Concept Foundation $25,000 2010 Membership Meeting IPPF, PAI, USAID/Ghana, USAID/Tanzania, Concept Foundation, FHI $44,000 Support to Secretariat staff CIES-UNAN/Nicaragua $2,000 Systems Strengthening WG: People that Deliver USAID|DELIVER, UNICEF, Global Fund, Project Optimize, Stop TB Partnership, CHAI, SCMS $260,000 TOTAL $10,665,000 2011 Scaling-up Concept Foundation’s QuRHM Project DFID $5,600,000 CARhs DFID $16,000,000 USAID $1,000,000 197 Evaluation of the Reproductive Health Supplies Coalition Access for All conference, Addis Ababa Bill & Melinda Gates Foundation $600,000 IPPF $5,000 Advance Family Planning project for regranting $250,000 LAC Forum Annual Meeting, Panama UNFPA/LACR $67,408 SECONAF Meeting, Dakar Hewlett Foundation $835 Subtotal (supplementary project support) $23,523,243 Cost savings from reductions in price of implants Implanon® (US$2 drop in price x 837,682 units sold at the new price). $1,675,362 Jadelle® (US$1.5 drop in price x 98,000 units sold at the new price). $147,000 Subtotal (new resources mobilized) $1,822,362 TOTAL $25,345,605 Indicator 3C.4 Total financial contribution of all institutions, per year, to sustaining Secretariat’s core operating costs Table 17: Financial contributions by institution for RHSC Secretariat core operations Year 2006 2007 2008 2009 2010 2011 Bill & Melinda Gates Foundation 853,000 1,480,000 1,300,000 565,000 655,000 870,000 DFID 60,000 567,000 790,000 UNFPA 494,000 392,000 0 USAID 0 380,000 420,000 Total $853,000 $1,480,000 $1,300,000 $1,119,000 $1,994,000 $2,080,000 Notes: Based on calendar year expenses. Numbers have been rounded. Does not include funds that have been received for additional activities. Indicator 3C.5 Total number of Coalition members and increase By the end of 2011, Coalition membership stood at 159 institutions—30 more than at the end of 2010. Of the new members, over one-third were from the Global South and they included four governments: China, Ethiopia, Senegal, and Zambia. As shown in Figure 4, the membership numbers for 2012 are already showing a visible upturn. The first four months saw 32 new members—more than in all of 2011. 198 Evaluation of the Reproductive Health Supplies Coalition 0 50 100 150 200 250 Figure 4. Growth in Coalition membership. 305968///01/01 30 June 2012 199 Evaluation of the Reproductive Health Supplies Coalition Appendix A: Key Indicators This table lists the indicators for each goal under the strategic areas, which were derived from the Strategic Plan’s focus areas and objectives. Indicators may be at country or global level under each of these areas; and where an indicator is wholly qualitative in nature, there may not be baseline statistics. Type Indicator 2007 2011 Units Responsibl e Partner Method of collection/ verification Goal 1. Increase availability, predictability, and sustainability of financing for RH Supplies Objectiv e 1A Mobilize additional new resources Indicato r 1A.1 Number of focus countries with a national budget line for RH supplies 6 13 Number Project RMA (assistance from UNFPA); DELIVER country annual reports Government reports; MOH or MOF verification of budget line item for RH supplies. Indicato r 1A.2 Percentage increase in donor funding for contraceptives in focus countries Value: $70,387,981 % change: 46.7% Value: $80,157,728 % change: 13.7% Number and % RHI for data – Secretariat for compilation Validated procured and paid for reproductive health supplies from Government records. If country is on RHI then shipped data can be used. Indicato r 1A.3 Percentage increase in donor funding for contraceptives worldwide Value: $182,819,24 3 % change: 37.6% Value: $219,168,17 3 % change: 8.2% Number and % RHI for data – Secretariat for compilation Data mining RHI database Indicato r 1A.4 Increase in number of donors making first-time contributions for RH supplies 7 11 Number UNFPA, Countdown 2015 Reports to UNFPA; Country level reports; Donor reports Indicato r 1A.5 Number of country proposals per year to the GFATM that make explicit provision for the purchase of RH supplies 3 countries procuring condoms with GF 31 countries procuring condoms with GF Number GFATM Data mining GFATM database (Available on website) or published procurement reports from partnership schemes Indicato r 1A.6 Increase in level of funding allocated to RH supplies by GFATM N/A N/A Number and % GFATM Data mining GFATM database (Available on website) or published procurement 305968///01/01 30 June 2012 200 Evaluation of the Reproductive Health Supplies Coalition Type Indicator 2007 2011 Units Responsibl e Partner Method of collection/ verification reports from partnership schemes Objectiv e 1B. Access existing resources Indicato r 1B.1 Proportion of focus country budget line(s) for RH supplies actually spent According to country According to country % Project RMA (assistance from UNFPA); DELIVER country annual reports Government reports; MOH or MOF verification of budget line item for RH supplies. Indicato r 1B.2 Examples of new domestic financing mechanisms for supplies proposed or introduced in focus countries. According to country No new domestic financing mechanism s found Narrative PSP-One, HPI Projects In country studies of available alternative financing [health insurance schemes, taxes, elimination of duties on RH supply donations to public sector, etc.] Objectiv e 1C. Mobilize Total Market Resources Indicato r 1C.1 Examples of new initiatives undertaken in focus countries to expand the total market for RH supplies According to country 2 TM Initiatives: Honduras and Madagascar Number Abt Associates, JSI, MSI, Futures Group and others involved in Total Market Initiative Market segmentation studies; PPP’s; Reports from partners Indicato r 1C.2 Increase in the non-public sector share of the market for condoms, pills and injectables in focus countries According to country According to country % share and increase DHS surveys; Social Marketing results (MSI, DKT, PSI and others) Market surveys undertaken by PSP-One partners, social marketing partners on sales and others Goal 2. Strengthen capacity of health systems to deliver RH supplies in a sustainable manner Objectiv e 2A. Strengthen existing systems Indicato r 2A.1 Number of focus countries with functioning RH commodity committees or 7 14 Number DELIVER, UNFPA Programme reports on existence and performance of commodity 305968///01/01 30 June 2012 201 Evaluation of the Reproductive Health Supplies Coalition Type Indicator 2007 2011 Units Responsibl e Partner Method of collection/ verification similar mechanisms for managing RH supply chains security committees or taskforces Indicato r 2A.2 Number of focus countries that include RH commodities on the national essential drug list 6 14 Number DELIVER, Secretariat Direct consultation of EDL – these are usually on-line Indicato r 2A.3 Examples of formal, coordinated actions by focus countries to improve RH supply chain performance (e.g. SPARHCS, national needs assessments, logistics systems strategies) TBD 11 countries including SPARHCS or RHSC strategies Number DELIVER; UNFPA; Crown Agents Programme reports from key partners assisting governments with supply chain performance. Country assessments Indicato r 2A.4 Number of RH products from specific producers/factorie s, pre-qualified under WHO/UNFPA Programme 14 46 Number WHO/ UNFPA WHO website and UNFPA website showing prequalified products by location Indicato r 2A.5 Number of all countries (and focus countries), per year, with active registered users of the RH Interchange 29 642 Number RHI RHI records of logon by country to the shipments section of the RHI database. Indicato r 2A.6 Decline in stock out rates at public-sector service delivery points in focus countries According to country According to country Number and % reduction DELIVER; UNFPA Quarterly data from country offices showing stock levels at Service Delivery Points. Indicato r 2A.7 Descriptions of private sector supply chains for RH supplies in the private sector and use in the According to country Bayer HealthCare and USAID’s "Contracep- tive Security Narrative/ Descriptio n DELIVER, PSI, MSI, DKT, IPPF Collection of already published descriptions/repor ts of private sector involvement in 305968///01/01 30 June 2012 202 Evaluation of the Reproductive Health Supplies Coalition Type Indicator 2007 2011 Units Responsibl e Partner Method of collection/ verification public sector Initiative" the supply chains for RH delivery. Indicato r 2A.8 Amount of funding provided through mechanisms that make RH supply resources less volatile (e.g., pledge guarantees) 0 $25,673,644 Number UNFPA for the AccessRH mechanism ; UNF for the Pledge Guarantee for Health mechanism ; Other mechanism s by Coalition partners Evidence that a focus country has accessed supplies utilising the appropriate funds Objectiv e 2B. To avert RH supply disruptions at national level - help when systems break down Indicato r 2B.1 Description of stockouts averted or resolved through the intervention of the Countries at Risk (CAR) Group N/A 168 supply crises addressed by CARhs Narrative CAR Group and Secretariat Minutes of the meetings/ Decisions taken and carried through. Objectiv e 2C. Systems for emergency situations Indicato r 2C.1 Number of Emergency RH kits sent to different countries by humanitarian agencies per year 8,990 8,534 Number UNFPA UNFPA database at Copenhagen procurement Division. Records at other humanitarian agencies Indicato r 2C.2 Examples of Coalition-inspired initiatives to integrate RH supplies into humanitarian relief efforts N/A MSI contraceptive security study Narrative UNFPA, MSI Findings from studies undertaken; reports of humanitarian efforts in RH supplies delivery Goal 3. Assure the added value of the RHSC as a productive and sustainable global partnership through efficiency, advocacy, and innovation Objectiv e 3A. Foster greater harmonization among stakeholders at global and country level Indicato r 3A.1 Examples of coordinated action by any two or more Coalition members, in the N/A Since 2009, six focus countries benefited Number and narrative RHSC Working Groups; Secretariat Narratives of instances from working groups, partners and country led 305968///01/01 30 June 2012 203 Evaluation of the Reproductive Health Supplies Coalition Type Indicator 2007 2011 Units Responsibl e Partner Method of collection/ verification same focus country, with the explicit aim of strengthening RH commodity security teams. Indicato r 3A.2 Examples of coordinated action by two or more Coalition institutions with the explicit aim of strengthening RH commodity security beyond the boundaries of a single country. N/A RHI, CARhs, PtD, LAC Forum, SECONAF Forum Number and narrative RHSC Working Groups; Secretariat Narratives of instances from working groups. Indicato r 3A.3 Examples of new Coalition tools and/or publications aimed at comparing or harmonizing comparable activities being implemented by two or more Coalition members. N/A Hand to Hand Campaign – new metric Narrative RHSC working groups; Secretariat Descriptions of tools developed by members of the working groups – or by consultants acting under Coalition initiatives Objectiv e 3B. Promote New Knowledge Indicato r 3B.1 Examples of all new tools, publications, and research initiatives produced each year, either by the Coalition or by its members in response to Coalition-inspired initiatives. 0 Country Profile’s on website, LAPTOP, Innovation Fund, RHSC publications Number and narrative RHSC Secretariat Routine collection of reports and initiatives from partners and within the Coalition Indicato r 3B.2 Annual increase in visits to the Coalition’s website N/A 1,738 avg. hits per month since June 2011 Number per month RHSC Secretariat Web-site hit figures produced by Active Elements (web- site host) Objectiv e 3C. Establish Support Systems Indicato 3C.1 Completion None Completed Report RHSC Analysis and 305968///01/01 30 June 2012 204 Evaluation of the Reproductive Health Supplies Coalition Type Indicator 2007 2011 Units Responsibl e Partner Method of collection/ verification r of Monitoring reports in 2009 and 2015, measuring core Coalition performance indicators against baseline values Complete d Secretariat evaluation of results of indicator changes Indicato r 3C.2 Number of institutions that contribute financially, each year, to the implementation of Coalition- sponsored or - inspired activities (excluding core support to Secretariat operations) 21 7 Number and names of contributor s RHSC Secretariat and all partners contributin g Consultation with Working Groups and Coalition members Indicato r 3C.3 Total financial contribution of all institutions, per year, to the implementation of Coalition- sponsored or - inspired activities (excluding core support to Secretariat operations) $1,286,401 $23,523,243 Value of contributio n RHSC Secretariat and all partners contributin g Consultation with Working Groups and Coalition members Indicato r 3C.4 Total financial contribution of all institutions, per year, to sustaining Secretariat’s core operating costs $853,000 $2,080,000 Amount in US$ PATH as the current grant holder From financial accounts of payments received Indicato r 3C.5 Total number of Coalition members and increase 14 159 Number Secretariat Membership records and active participation in Coalition activities through being members of the at least one working group TBD = To be determined N/A = Not Applicable 305968///01/01 30 June 2012 205 Evaluation of the Reproductive Health Supplies Coalition

View the publication

You are currently offline. Some pages or content may fail to load.