External evaluation 2008-2012: advancing sexual and reproductive health

Publication date: 2013

External evaluation 2008–2012 Advancing sexual and reproductive health WHO/RHR/HRP/13.07 Contents Detailed contents and page numbers are provided within each chapter Chapter 1. Introduction . 1 Nicholas Dodd Consultant, Sexual and Reproductive Health Chapter 2. Overall assessment of HRP’s relevance and effectiveness . 23 Nicholas Dodd Consultant, Sexual and Reproductive Health Chapter 3. Efficiency and effectiveness of HRP’s governance, management and administration . 71 Nicholas Dodd Consultant, Sexual and Reproductive Health Chapter 4. Evidence generation and synthesis to improve family planning, prevent unsafe abortion and prevent and control sexually transmitted and reproductive tract infections . 101 Affette McCaw‐Binns Professor of Reproductive Health and Epidemiology and Head, Department of Community Health and Psychiatry, University of the West Indies, Jamaica Jasneth Mullings Research Fellow, Department of Community Health and Psychiatry, University of the West Indies, Jamaica Chapter 5. Research‐capacity strengthening and network building . 153 Chisale Mhango Senior Lecturer, Department of Obstetrics and Gynaecology, College of Medicine, Blantyre, Malawi Chapter 6. Strengthening implementation research . 183 Fernando Althabe Director, Department of Mother and Child Research Institute for Clinical Effectiveness and Health Policy in Buenos Aires, Argentina Daniela S Colaci Researcher, Department of Mother and Child Research Institute for Clinical Effectiveness and Health Policy in Buenos Aires, Argentina Chapter 7. The status of, and opportunities for strengthening, engagement with the private sector and civil society . 219 International Development Team Pricewaterhouse Coopers SA . Geneva, Switzerland Chapter 8. Bibliometric analysis . 245 Karen Gurney Manager, Bibliometric Reporting, Thomson Reuters (Evidence), Leeds, United Kingdom of Great Britain and Northern Ireland Bethan Sheridan-Jones Data Analyst, Thomson Reuters (Evidence), Leeds, United Kingdom of Great Britain and Northern Ireland Simon Thomson Data Analyst, Thomson Reuters (Evidence), Leeds, United Kingdom of Great Britain and Northern Ireland Page | 1 Chapter 1 Introduction Nicholas Dodd Consultant, Sexual and Reproductive Health Page | 2 Page | 3 Contents 1 Background to the Special Programme of Research, Development and Research Training in Human Reproduction . 5 2 Previous external evaluations of HRP . 6 3 The current external evaluation of HRP . 7 References . 9 Annex 1: Terms of reference for the 2008–2012 external evaluation of HRP . 11 Background . 11 Proposal for HRP external evaluation 2008–2012 (to be presented in June 2013) . 13 Annex 2: the Memorandum of Understanding of HRP . 16 Page | 4 Abbreviations HRP UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction ICPD International Conference on Population and Development IEGWB Independent Evaluation Group of the World Bank MDG Millennium Development Goal MSH Management Sciences for Health OED Operations Evaluation Department (World Bank) PCC Policy and Coordination Committee PDRH Programme Development in Reproductive Health PEEC PCC External Evaluation Committee RHR WHO Department of Reproductive Health and Research SCIH Swiss Centre for International Health SRHR sexual and reproductive health and rights STAG Scientific and Technical Advisory Group UN United Nations UNDP United Nations Development Programme UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund WHO World Health Organization HRP External Evaluation 2008−2012 Chapter 1 Page | 5 1 Background to the Special Programme of Research, Development and Research Training in Human Reproduction The Special Programme of Research, Development and Research Training in Human Reproduction (HRP) was established by the World Health Organization (WHO) in 1972 to coordinate, promote, conduct and evaluate international research in human reproduction. Sixteen years later, in 1988, HRP became a cosponsored programme, with an explicit mandate for: • promoting and supporting research aimed at finding and developing safe and effective methods of fertility regulation, and identifying and eliminating obstacles to such research and development; • identifying and evaluating health and safety problems associated with fertility regulation technology, analysing the behavioural and social determinants of fertility regulation, and testing cost-effective interventions to develop improved approaches to fertility regulation within the context of reproductive health services; • strengthening the training and research capability of developing countries in the field of human reproduction; • establishing a basis for collaboration with other programmes engaged in research and development in human reproduction, which will include the identification of priorities across the field and the coordination of activities in the light of such priorities (1). The original four cosponsors were the United Nations Development Programme (UNDP), the United Nations Population Fund (UNFPA), the World Bank, and WHO, and the Memorandum of Understanding agreed to by these parties (see Annex 2) remains today as the document that guides HRP’s overall purpose as well as its governance. HRP is the only body within the United Nations (UN) system mandated to lead research in human reproduction. It works in close association with countries to provide the answers to critical sexual and reproductive health questions, resulting in the generation of evidence-based research findings and the information needed to achieve universal access to effective services and to enable people to protect and promote their own sexual and reproductive health. Since 1998, HRP has functioned as one of the two components in the Department of Reproductive Health and Research (RHR). The second component is Programme Development in Reproductive Health (PDRH), whose function became to facilitate the application of research results to policy and practice in sexual and reproductive health programmes in countries, thereby raising the value of HRP’s outputs, and increasing the effectiveness of WHO’s work in sexual and reproductive health. HRP’s modus operandi remains largely unchanged since its inception. It develops 5-year strategic plans and biennial workplans and budgets, to implement a prioritized global research agenda, and to provide support to strengthen research capacity in institutions in programme countries. Its budgetary support for these two items remains at 2:1, respectively for research and for capacity strengthening. Its proposed programme of work is reviewed by its Scientific and Technical Advisory Group (STAG), and its Policy and Coordination Committee (PCC) provides oversight for funding, policy and programme management. The content of HRP’s work, however, has changed substantially over the years, from an early focus on developing new methods of fertility regulation and clinical studies on the safety and efficacy of new and existing methods of fertility regulation, to an agenda focused on the broader spectrum of sexual and reproductive health, including: family planning; maternal and perinatal health; sexually transmitted and reproductive tract infections; preventing unsafe HRP External Evaluation 2008−2012 Chapter 1 Page | 6 abortion; and gender, reproductive rights, sexual health and the sexual and reproductive health of adolescents; as well as research on implementation of the sexual and reproductive health products and tools the Programme produces. HRP’s overriding vision is: . . . the attainment by all peoples of the highest level of sexual and reproductive health. We strive for a world where all women’s and men’s rights to enjoy sexual and reproductive health are promoted and protected, and all women and men, including adolescents and those who are underserved and marginalized, have access to sexual and reproductive health information and services (2). Over the last 20 years, the global landscape on sexual and reproductive health has progressively developed. HRP was a key resource to these processes and then incorporated their outcomes fully into its work. They included: the Programme of action of the 1994 International Conference on Population and Development (ICPD) in Cairo (3), and its various follow-up mechanisms; the Beijing Declaration and platform for action of the Fourth World Conference on Women in 1995 (4), which reaffirmed the reproductive health agenda, and its follow-up mechanisms; the Millennium Development Goals (MDGs) (5), including the new target 5B, on universal access to reproductive health; the UN Secretary-General’s Global strategy for women’s and children’s health (6); the WHO Global reproductive health strategy, adopted at the World Health Assembly in 2004 (7); the report of the WHO Commission on the Social Determinants of Health (8), which focused on health inequities; the WHO Global strategy for the prevention and control of sexually transmitted diseases: 2006 – 2015 (9); and, most recently, the 2012 London Family Planning Summit. Many regional commitments also ensued, for example the Maputo Plan of action on sexual and reproductive health and rights (10) for a continental policy framework for sexual and reproductive health and rights in Africa. By both helping to guide global change and then adapting its work to such change, HRP has continued to demonstrate that its business model functions very well, and that it remains relevant to the needs of programme countries. As well as conducting research, and building capacity to conduct research, the Programme also synthesizes research through systematic reviews of the literature, and develops tools that facilitate access, by countries and by individuals, to the latest research information. Research guides policies and programmes, and nowhere is this more important than in the area of sexual and reproductive health. Without research, policies become obsolescent, and information and service programmes do not provide people with the latest information, or the best technologies and tools to ensure their sexual and reproductive health. 2 Previous external evaluations of HRP HRP has a long-standing culture of regularly submitting its work and functioning to external evaluations, of which the last two are briefly reviewed here. In 2002, an external evaluation, jointly conducted by Management Sciences for Health (MSH) and the Swiss Centre for International Health (SCIH) of the Swiss Tropical Institute, focused on four key issues (10). These were: the relevance and effectiveness of HRP-supported research in reproductive health; the dissemination, global use and impact of the results of HRP’s reproductive health research; reproductive health research-capacity strengthening by HRP and the use and impact of HRP’s work at country level; and the HRP governance process, management, administration and efficiency. Two thematic case-studies were also undertaken, the first on emergency contraception and the second on mainstreaming gender and women’s perspectives. The evaluation examined HRP’s work between 1990 and 2002. HRP External Evaluation 2008−2012 Chapter 1 Page | 7 The overall conclusion of this evaluation was that: HRP clearly met expectations in terms of its core mission to coordinate, promote, conduct and evaluate international research in reproductive health and achieved its major objectives. The Programme maintained its position as the global leader in generating research results and establishing the scientific consensus needed to advance reproductive health policies and practices, especially for developing countries (11). The most recent external evaluation reviewed the period 2003 to 2007 (12), and was conducted by a team of eight international experts during the course of 2006–2007. The overall focus of the evaluation was on the impact of the Programme on global public goods, and this was examined in more detail through five case-studies on: the long-term safety and effectiveness of the copper-releasing intrauterine devices; improving the quality of family planning care in China; medical (non-surgical) abortion; improving maternal and newborn health; and knowledge synthesis and transfer. In addition, the evaluation also reviewed HRP’s follow-up actions to the recommendations of the previous evaluation in the areas of governance, management, administration and efficiency. The overall conclusion of the 2003–2007 evaluation was that: HRP remains a global leader in sexual and reproductive health research and capacity building, with particular relevance to the needs of populations in resource-poor settings. The evidence base resulting from this research has been translated effectively into health policy changes and improved practice standards and has ultimately improved health outcomes. 3 The current external evaluation of HRP The current evaluation was requested by the World Bank at the 71st meeting of the standing committee in June 2011. At this meeting, the cosponsors agreed on draft terms of reference, elaborating an approach that would review the comparative advantage of HRP and its impact in improving outcomes and influencing evidence-based changes in sexual and reproductive health policies and programmes, as well as carrying out a number of case-studies. The standing committee also recommended the establishment of a PCC External Evaluation Committee (PEEC) to include: the chair and vice-chair of PCC at its 24th meeting on 16–17 June 2011, one representative of the HRP financial contributors, the chair of RHR’s STAG, and one representative of the four HRP cosponsors, in order to oversee the process of the evaluation. Terms of reference for the evaluation were subsequently shared with PCC members for feedback, and finalized at the 72nd meeting of the standing committee in December 2011. The current evaluation of HRP covers the period 2008 to 2012, and the full terms of reference can be found in Annex 1. The evaluation reviewed HRP’s overall relevance and effectiveness, particularly in terms of producing global public health goods, and the efficiency and effectiveness of its governance, management and administration (Nicholas Dodd, who also acted as overall coordinator of the evaluation), as well as conducting the four case-studies. These examine the following issues: evidence generation and synthesis to improve family planning, prevent unsafe abortion and prevent and control sexually transmitted diseases and reproductive tract infections (Affette McCaw-Binns and Jasneth Mullings, Chapter 4); research- capacity strengthening and network building (Chisale Mhango, Chapter 5); strengthening implementation research (Fernando Althabe and Daniela Colaci, Chapter 6); and the status of, and opportunities for strengthening, engagement with the private sector and civil society (International Development Team, Pricewaterhouse Coopers SA, Chapter 7). HRP External Evaluation 2008−2012 Chapter 1 Page | 8 The evaluation was carried out through a combination of desk reviews of documentation; interviews with key informants; questionnaires; and site visits. The methodology for the evaluation tried to follow the example of the Programme itself. It attempted to be systematic, and to use an evidence-based approach, relying on quantitative data, supplemented wherever possible by qualitative information as appropriate. Further details of the specific methodologies used for each section of the evaluation, as well as findings, conclusions and recommendations, are to be found in the respective chapters. The evaluation also included the commissioning of a bibliometric analysis, which is discussed further in Chapter 2 of the evaluation report. In June 2012, at the time of the 25th meeting of PCC, the majority of the evaluation team was able to travel to Geneva to have direct consultations with HRP staff, to plan and strategize for the evaluation among themselves, to develop and agree on a timeline, and to interact with the PCC subcommittee (PEEC) charged with overseeing the evaluation, as well as with the wider group of PCC members and observers. At this time, an e-mail was also sent to all RHR staff, inviting their input and suggestions, and, as a result, a number of staff members submitted information on their work and ideas for the evaluation process. A timeline for the evaluation was discussed and agreed, and the consultants subsequently shared chapter outlines for discussion and review in mid-July, and then started working on the content of their respective assignments. Progress reports on the evaluation were submitted to PEEC in August and November 2012. An oral presentation of the evaluation report was made at the 30th meeting of the STAG in February 2013. The full report was subsequently reviewed by PEEC and the standing committee in April 2013 and presented to PCC at its 26th meeting in June 2013. Acknowledgement The evaluation team would like to thank most sincerely all those who gave freely of their time and their knowledge in the process of preparing this report. These include current and former staff of HRP; current and former staff of the cosponsors; current and former members of HRP’s governance committees; directors and staff of research institutions in programme countries that have received support from HRP; all those who responded to questionnaires, including national directors of sexual and reproductive health in programme countries, UNFPA country offices, and numerous civil society organizations. The report benefited enormously from your help and your ideas. Finally, we would like to thank Mike Mbizvo, the outgoing Director of HRP, for his guidance, patience, and constant availability whenever questions needed to be answered, information needed to be provided or advice needed to be sought. HRP External Evaluation 2008−2012 Chapter 1 Page | 9 References 1. Memorandum on the administrative structure of the Special Programme of Research, Development and Research Training in Human Reproduction (HRP). Geneva, World Health Organization, 2012 (http://www.who.int/hrp/governance/HRP_MoU_Admin_Structure_rev_2012.pdf, accessed 24 December 2012). 2. WHO’s strategic vision in sexual and reproductive health and rights. Business Plan 2012–2015. Geneva, World Health Organization, 2010 (WHO/RHR/10.10) 3. UNFPA. Programme of action. Adopted at the International Conference on Population and Development, Cairo, 5–13 September 1994. New York, United Nations, 1995 (http://www.unfpa.org/public/home/publications/pid/1973, accessed 19 December 2012). 4. Beijing Declaration and platform for action of the Fourth World Conference on Women in 1995. New York United Nations, 1996 (http://www.un.org/womenwatch/daw/beijing/pdf/BDPfA%20E.pdf, accessed 19 December 2012). 5. UNDP. Millennium Development Goals (http://www.undp.org/content/undp/en/home/mdgoverview.html, accessed 19 December 2012). 6. United Nations Secretary-General. Global strategy for women’s and children’s health. New York, United Nations, 2010 (http://www.who.int/pmnch/topics/maternal/201009_globalstrategy_wch/en/index.h tml, accessed 19 December 2012). 7. World Health Organization, Department of Reproductive Health and Research. Reproductive health strategy: to accelerate progress towards the attainment of international development goals and targets. Geneva, World Health Organization, 2004 (WHO/RHR/04.8) (http://whqlibdoc.who.int/hq/2004/WHO_RHR_04.8.pdf, accessed 19 December 2012). 8. Commission on Social Determinants of Health. Report by the Secretariat. Geneva, World Health Organization, 2008 (http://apps.who.int/gb/ebwha/pdf_files/EB124/B124_9-en.pdf, accessed 19 December 2012). 9. World Health Organization, Department of Reproductive Health and Research. Global strategy for the prevention and control of sexually transmitted diseases: 2006–2015. Breaking the chain of transmission. Geneva, World Health Organization, 2007 (http://www.who.int/reproductivehealth/publications/rtis/9789241563475/en/, accessed 19 December 2012). 10. Plan of action on sexual and reproductive health and rights (Maputo Plan of Action). Addis Ababa, The African Union Commission, 2006 (http://www.unfpa.org/africa/newdocs/maputo_eng.pdf, accessed 19 December 2012). HRP External Evaluation 2008−2012 Chapter 1 Page | 10 11. UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Reproduction. External evaluation 1990–2002. Executive summary. Geneva, World Health Organization, 2003 (http://whqlibdoc.who.int/hq/2003/WHO_RHR_HRP_03.14.pdf, accessed 19 December 2012). 12. UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Reproduction. External evaluation 2003–2007: executive summary. (http://www.who.int/reproductivehealth/publications/reports/RHR_08_4/en/index.ht ml, accessed 20 December 2012). 13. IEGWB Guidelines for global program reviews (http://www.worldbank.org/ieg/grpp/docs/gprguidelines.pdf, accessed 20 December 2012). 14. OECDE DAC Network on Development Evaluation. Evaluating development co- operation: summary of key norms and standards, 2nd ed. Paris, OECD, 2010 (http://www.oecd.org/development/evaluationofdevelopmentprogrammes/dcdndep/ 41612905.pdf, accessed 19 December 2012). HRP External Evaluation 2008−2012 Chapter 1 Page | 11 Annex 1: Terms of reference for the 2008–2012 external evaluation of HRP Background The United Nations Development Programme (UNDP)/United Nations Population Fund (UNFPA)/World Health Organization (WHO)/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP) was established in 1972, and became a cosponsored United Nations (UN) Special Programme in 1988, with a explicit mandate, as endorsed by the 41st World Health Assembly (1988) in resolution 41.9, for: (1) the continued assessment of existing technologies and the acceleration of the development of new technologies in fertility regulation; (2) the building-up of national self-reliance in research on all aspects of human reproduction in developing countries to meet their specific needs in primary health care; (3) promoting scientific and technical cooperation between developed and developing countries, and between developing countries; (4) coordination of the global research effort in the field of reproductive health; (5) promoting ethical practices in the field of human reproduction research to protect the health and rights of individuals in different social and cultural settings. Over the years, the work of HRP has evolved, in order to best respond to this mandate and adapt to evolving needs, global context and commitments. In order to ensure its effectiveness and efficiency in carrying out this mandate, HRP has been subject to periodic independent external evaluations, commissioned by its Policy and Coordination Committee (PCC). These evaluations are carried out in order to ensure the efficiency and accountability of HRP, as well as to advocate for its work and respond to specific requirements of its donors and cosponsors. HRP external evaluation 1990–2002 (presented in 2003) In 2002, a comprehensive evaluation was conducted, covering the period 1990–2002.This evaluation was considered by PCC at its 16th meeting on 30 June to 1 July 2003. The executive summary is available at http://whqlibdoc.who.int/hq/2003/WHO_RHR_HRP_03.14.pdf (10). The 1990–2002 external evaluation was conducted by Management Sciences for Health (MSH) and the Swiss Centre for International Health (SCIH) of the Swiss Tropical Institute. These organizations, working as a consortium, were selected following an international tender process by the External Evaluation Monitoring Team (EEMT), set up by PCC to select the external evaluators, to provide overall guidance to the external evaluation and, in particular, to ensure that the external evaluation report fully addressed the terms of reference given to the external evaluation team. The 1990–2002 external evaluation was a wide-ranging, comprehensive study designed to address four key issues: (1) the relevance and effectiveness of Programme-supported research in reproductive health; (2) the dissemination, global use and impact of the results of the Programme's reproductive health research; (3) reproductive health research-capacity strengthening by the Programme and the use and impact of the Programme's work at country level; and (4) the Programme's governance process, management, administration and efficiency. Conclusions and recommendations made by the external evaluation team were based on document review, citation analysis of selected publications, seven country visits, and HRP External Evaluation 2008−2012 Chapter 1 Page | 12 input from more than 300 informants, of whom 249 provided detailed information through face-to-face interviews and e-mail questionnaires. Two thematic case-studies (one on emergency contraception and one on mainstreaming gender and women's perspectives) were also performed, which provided further in-depth information on specific aspects of the Programme's work. The external evaluation report provided a strong and favourable endorsement of the direction and management of the Programme. The overall conclusion of the external evaluation, as reported in the evaluation report, was that, during the period 1990–2002: HRP clearly met expectations in terms of its core mission to coordinate, promote, conduct and evaluate international research in reproductive health, and achieved its major objectives. The Programme maintained its position as the global leader in generating research results and establishing the scientific consensus needed to advance sexual and reproductive health policies and practices, especially for developing countries (12). The external evaluation also made numerous recommendations, described in the report, to further enhance the performance of the Programme. The Programme reported to PCC on implementation of these recommendations at its 17th meeting on 30 June to 1 July 2004. HRP external evaluation 2003–2007 (presented in 2008) In 2003, the HRP standing committee and PCC asked for an evaluation focusing specifically on the impact of the Programme on global public goods. Public goods are generally defined as those goods that: produce benefits that are non-rival (many people can consume, use, or enjoy the good at the same time) and non-excludable (it is difficult to prevent people who do not pay for the good from consuming it). If the benefits of a particular good accrue across all or many countries, then this is deemed a global or international public good (13). The International Task Force on Global Public Goods has made the above definition operational as follows: International public goods, global and regional, address issues that: (i) are deemed to be important to the international community, to both developed and developing countries; (ii) typically cannot, or will not, be adequately addressed by individual countries or entities acting alone, and, in such cases (iii) are best addressed collectively on a multilateral basis (13). In terms of its mandate and the nature of its outputs, as well as with respect to its modus operandi, the Programme is, without doubt, a major contributor to global public goods; thus, this was the suggested focus of the evaluation. HRP was established by WHO in 1972 to coordinate, promote, conduct and evaluate international research in human reproduction. The UNDP, the UNFPA and the World Bank joined WHO as cosponsors of HRP in “coordination of the global research effort in the field of reproductive health”. As the main instrument within the UN system for research in human reproduction, HRP brings together health-care providers, policy-makers, scientists, clinicians and consumer and community representatives to identify and address priorities for research aimed at improving sexual and reproductive health. Since 1998, HRP has functioned within the WHO Department of Reproductive Health and Research (RHR). HRP investigates the extent and nature of sexual and reproductive health problems, their determinants and the interventions needed for their alleviation or resolution. Its research agenda addresses all of the main challenges in sexual and reproductive health identified in HRP External Evaluation 2008−2012 Chapter 1 Page | 13 international fora, particularly the International Conference on Population and Development (ICPD) in 1994 and the Fourth World Conference on Women in 1995, and their respective 5- year follow-ups. HRP also carries out activities to strengthen the capabilities of low- and middle-income countries to meet their own research needs, and to enable them to participate in global sexual and reproductive health research. HRP promotes the use of research results in policy-making and planning at national and international levels and contributes to the setting of norms, standards and guidelines – including ethical guidelines – in the field of sexual and reproductive health research. In order to foster the achievement of greater equity and reproductive rights, HRP works to ensure that gender issues, especially the perspectives of women, are reflected in both its research and research capability strengthening activities. The evaluation was conducted by a team of global experts selected for each case-study and focused on five Programme achievements that fulfil the criteria of global public goods and lent themselves to an in-depth analysis of inputs, outputs, outcomes and, where possible, impact on sexual and reproductive health status and contribution to achievement of MDGs, including poverty alleviation. The five technical case-studies were: (1) promoting family planning: long- term safety and effectiveness of copper-releasing intrauterine devices; (2) promoting family planning: improving the quality of care in family planning in China; (3) medical (non-surgical) induced abortion; (4) improving maternal and newborn health; and (5) knowledge synthesis and transfer. In addition, a study of HRP’s governance and management was carried out. The evaluation concluded that: HRP remains a global leader in sexual and reproductive health research and capacity building, with particular relevance to the needs of populations in resource-poor settings. The evidence base resulting from this research has been translated effectively into health policy changes and improved practice standards and has ultimately improved health outcomes. The case-studies indicate that HRP is in a good position to continue advancing global public goods in a cost-effective way (12). The resulting evaluation summary and the six case-studies are available at http://www.who.int/reproductivehealth/publications/reports/RHR_08_4/en/index.html (12). Proposal for HRP external evaluation 2008–2012 (to be presented in June 2013) The 2008–2012 HRP external evaluation will aim to provide information on (1) the relevance and fulfilment of HRP's objectives; (2) its efficiency and effectiveness; (3) its comparative advantage; and (4) the impact and sustainability of its work. In doing so, it will provide information that is credible and will enable the continued incorporation of lessons learnt into the decision-making process of both the Special Programme and its constituents. In view of the positive feedback received from PCC on the process followed in the previous evaluation, it is proposed to follow a similar process in 2012–2013, in order to examine in depth a number of the critical programme areas and outputs that have not been examined recently. Method of work It is envisaged that the external evaluation report would consist of an introductory, overview chapter followed by seven chapters that would provide in-depth studies of the selected topics. The introductory chapter would give the background to the Programme as well as a brief overall assessment of the Programme’s contribution to global public good as it relates to sexual and reproductive health and rights (SRHR). It would also evaluate the Programme’s modus operandi, including its governance and the consultative mechanisms it employs for HRP External Evaluation 2008−2012 Chapter 1 Page | 14 defining and prioritizing its work programmes and for the scientific and ethical review of planned, ongoing and completed activities. Furthermore, it would also describe the overall frame of reference that the Programme uses in pursuing its mission and vision, and thus would address such issues as the UN Secretary-General’s Global strategy for women’s and children's health (5), the Millennium Development Goals (MDGs), including poverty reduction (4), the Global reproductive health strategy (6), and the means by which it collects, manages and applies evidence to improve sexual and reproductive health in countries. The chapter would specifically address the following issues: 1. a brief overall assessment of the Programme's contribution to global public good, as well as its comparative advantage, as it relates to SRHR, in terms of its mandate, modus operandi and output. This assessment will include considerations on the uptake and utilization of HRP guidance, tools and strategic approaches: • does HRP continue to provide leadership on SRHR global health matters and to be important to the international community, to low-, middle- and high-income countries? • does it address key SRHR issues that will not typically be adequately addressed by individual countries or entities acting alone? • does it shape the SRHR-HIV research agenda, reflecting key programmatic and policy questions and needs? • does it set norms and standards, and articulate evidence-based policy and practices at the national and global levels? • does it monitor and assess effectively reproductive health trends, and conduct important evaluation? • does it identify, promote and evaluate innovative strategies and technologies? • does it communicate effectively within and beyond WHO? The in-depth studies will focus on the following areas of major interest in relation to the Programme’s mandate and plan of work: 2. overall efficiency and effectiveness of HRP’s governance, management and administration; 3. evidence generation and synthesis to improve family planning, prevent unsafe abortion and prevent and control sexually transmitted and reproductive tract infections; 4. research-capacity strengthening and network building; 5. strengthening implementation research; 6. the status of and opportunities for strengthening engagement with the private sector and civil society. An executive summary will be prepared in order to share the findings in a concise manner among stakeholders, potential partners and other interested parties. It is anticipated that most of the external evaluation can be conducted through desk review of materials, and consultation, by tele- and videoconferencing and other means, with relevant programme staff and key respondents in other organizations (multilateral and bilateral agencies; professional and other nongovernmental organizations), scientists and programme managers at national level, etc. Nevertheless, for each case-study, funds are included in the HRP External Evaluation 2008−2012 Chapter 1 Page | 15 budget to support one visit (3 days) to the Programme in Geneva and one country visit, if deemed essential. It is proposed that each study would be carried out by an independent expert in the field, who would be nominated by and report back to a subcommittee of PCC, entitled the PCC External Evaluation Committee (PEEC). PEEC is comprised of the PCC chair and/or vice chair, the STAG chair and at least two representatives of the standing committee. The logistics of the evaluation, including travel and other arrangements, would be supported by the secretariat. The evaluation will be carried out in accordance with Evaluating development co-operation: summary of key norms and standards (14), which was issued in 2010 by the Network on Development Evaluation of the Development Assistance Committee of the Organisation for Economic Co-operation and Development. HRP External Evaluation 2008−2012 Chapter 1 Page | 16 Annex 2: the Memorandum of Understanding of HRP HRP External Evaluation 2008−2012 Chapter 1 Page | 17 HRP External Evaluation 2008−2012 Chapter 1 Page | 18 HRP External Evaluation 2008−2012 Chapter 1 Page | 19 HRP External Evaluation 2008−2012 Chapter 1 Page | 20 HRP External Evaluation 2008−2012 Chapter 1 Page | 21 HRP External Evaluation 2008−2012 Chapter 1 Page | 22 Page | 23 Chapter 2 Overall assessment of HRP’s relevance and effectiveness Nicholas Dodd Consultant, Sexual and Reproductive Health Page | 24 Page | 25 Contents 1 Introduction . 29 1.1 A note on language . 29 2 Methods . 29 3 Recent challenges . 30 4 The questionnaire . 31 5 Global public health goods . 33 5.1 Priority setting . 33 5.2 Geographical focus . 37 5.3 Coordination of research . 39 5.4 Selected highlights . 40 6 Comparative advantage . 51 6.1 Respondents’ views on HRP’s comparative advantage . 52 7 Norms and standards . 53 8 Monitoring of global trends in sexual and reproductive health . 54 8.1 Global maternal mortality estimates. . 54 8.2 Global estimates of unsafe abortion . 55 9 The sexual and reproductive health/HIV research agenda . 56 9.1 Selected highlights . 56 10 Efficiency and effectiveness of HRP’s communication . 58 10.1 Bibliometric analysis of HRP’s papers . 60 10.2 HRP’s channels of communication . 61 10.3 Changes in RHR over the last 5 years and how these have affected HRP’s research communication and uptake . 62 References . 66 Page | 26 Abbreviations BTN “Beyond the numbers” CDC Centers for Disease Control and Prevention CHNRI Child Health and Nutrition Research Initiative CIRE Continuous Identification of Research Evidence CSO civil society organization CVC Core Voluntary Contribution DMT Decision-making tool for family planning clients and providers EPC “Effective perinatal care” FGM female genital mutilation GAP Gender and Rights Advisory Panel GBV gender-based violence GCP good clinical practice GRR WHO Gender and Reproductive Rights team GTZ German Development Agency HPV human papillomavirus HRP UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction IPPF International Planned Prenthood Federation MCA Department of Maternal, Newborn, Child and Adolescent Health MDG Millennium Development Goal MEC Medical eligibility criteria for contraceptive use MPH Improving Maternal and Perinatal Health (team) MTCT mother-to-child transmission MTSP medium-term strategic plan NASG non-pneumatic anti-shock grament NGO nongovernmental organization OHCHR Office of the High Commissioner for Human Rights PEEC PCC External Evaluation Committee PCC Policy and Coordination Committee PDRH Programme Development in Reproductive Health PPH postpartum haemorrhage RAP regional advisory panel RHL Reproductive Health Library (WHO) RHR WHO Department of Reproductive Health and Research RP2 Research Project Review Panel Page | 27 RTI reproductive tract infection SPP WHO/UNFPA Strategic Partnership Programme SRH sexual and reproductive health STAG Scientific and Technical Advisory Group STI Sexually transmitted infection TDR Special Programme for Research and Training in Tropical Diseases UK United Kingdom of Great Britain and Northern Ireland UN United Nations UNAIDS Joint United Nations Programme on HIV/AIDS UNDP United Nations Development Programme UNECA United Nations Economic Commission for Africa UNESCO United Nations Education, Scientific and Cultural Organization UNFPA United Nations Population Fund UNHCR United Nations High Commissioner for Refugees UNICEF United Nations Children’s Fund UNIFEM United Nations Development Fund for Women USA United States of America USAID United States Agency for International Development WHO World Health Organization Page | 28 HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 29 1 Introduction This chapter reviews the overall relevance and effectiveness of the United Nations Development Programme (UNDP)/United Nations Population Fund (UNFPA)/United Nations Children’s Fund (UNICEF)/World Health Organization (WHO)/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP); its research; its comparative advantage and the value added by its location within WHO; its work on norms and standards; its monitoring of global trends in sexual and reproductive health (SRH); its work related to the SRH/HIV research agenda; and how it communicates its products to its clients. 1.1 A note on language HRP will also be referred to throughout this document as “the Programme”; those countries that are the main intended beneficiaries for HRP’s work will be collectively referred to as “programme countries”; and the various outputs of the Programme, whether they be physical devices or printed materials, will be collectively referred to as “the products” of the Programme. Finally, the term “sexual and reproductive health” is implicitly assumed to always include a component of sexual and reproductive rights. 2 Methods The process for the evaluation began in Geneva, with preliminary consultations and interviews with senior staff of the Programme, as well as WHO staff in other related departments including: the Family and Community Health Cluster; HIV Department; the Special Programme for Research and Training in Tropical Diseases (TDR); the Global Management Cluster; the Partnership for Maternal, Newborn and Child Health; and, the Alliance for Health Policy and Systems Research. The evaluation team also met with the subcommittee of the Policy and Coordination Committee (PCC) charged with overseeing the evaluation (PCC External Evaluation Committee – PEEC). In addition, an e-mail was sent to all staff of the WHO Department of Reproductive Health and Research (RHR), inviting their input and suggestions, and as a result a number of staff members submitted information on their work and ideas for the evaluation process. This was followed by a review of all relevant documentation produced by the Programme over the past 5 years. For Chapters 2 and 3 of the evaluation – the overall assessment of HRP and its governance, administration and management – a 65-point questionnaire was then developed and sent out in full confidence to over 400 key informants. Informant categories included the following: directors of national SRH programmes; PCC members; PCC observers; HRP technical committee members (the Scientific and Technical Advisory Group (STAG); the Gender and Rights Advisory Panel (GAP); regional advisory panels (RAPs); and the Research Project Review Panel (RP2)); current and former headquarters, and regional and country staff of the cosponsors; organizations with an interest either in carrying out SRH research or in using the results of such research; and individual experts in the field of SRH. Although there was insufficient time or resources for full pretesting of the questionnaire, it was reviewed by a number of experts before being distributed. Subsequently, a much shorter questionnaire on the relevance and the use of HRP’s products in national SRH programmes was sent to all UNFPA country offices. The rationale here was that, of all the cosponsors, UNFPA’s mandate in terms of substance coincides most directly with the mandate of HRP. Thus, UNFPA would be in a good position to identify examples of the utilization of HRP’s products in countries. The results are reported in Sections 5 and 10. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 30 A citation analysis was also commissioned as part of the evaluation and the results of this are also summarized in Section 10.1. As the evaluation progressed, it became apparent that it was not always easy to distinguish between the outputs of HRP and the outputs of the Programme Development in Reproductive Health (PDRH). Publications citing achievements of HRP often appeared to include work carried out by PDRH. Operationally, the relationship between HRP and PDRH is very important; in essence, HRP does the research and normative work, and PDRH takes HRP’s products and promotes and applies them in countries through various mechanisms. However, in terms of monitoring and evaluating the work of these two arms of RHR, the distinction between who does what needs to be made clearer. Recommendation HRP needs to clearly identify in its reporting mechanisms the results it achieves, as distinct from the results achieved by PDRH. 3 Recent challenges Every fund, agency or programme goes through periods of instability and change. The way in which they respond is often a test of their fundamentals. Over the past five years, HRP has faced a number of challenges including: • a substantial delay from January 2009 until December 2010 in the appointment of Dr Paul van Look’s successor as director of HRP; • the WHO hiring freeze at a time when a larger than normal cohort of senior HRP staff were retiring; • introduction of a new process for the allocation of WHO core budget resources; • introduction of the new WHO ORACLE management system and the decentralization of some WHO administrative functions; • the proposal to rationalize and possibly aggregate a number of WHO research programmes; • the WHO decision to move Making Pregnancy Safer, formerly part of RHR, not back into RHR but as part of the Department of Maternal, Neonatal, Child and Adolescent Health; • major changes in the long and well-established HRP process for ethical review; • the increasing complexity of the global health architecture and HRP’s need to both adapt to, and service, parts of this new and changing architecture; • the fall of the US dollar against the Swiss Franc; • continuing pressure for HRP to take on a number of new areas of work or increase its work in certain areas – such as research on gender, sexual and reproductive rights, sexuality, adolescents, SRH in emergency and crisis situations, the cost effectiveness of SRH, and the SRH of sexual minorities and people with disabilities, and implementation research, to name but a few. All of these occurred without a corresponding increase in funds, and amidst a global financial crisis followed by the Euro crisis, which led donor countries, foundations and cosponsors to review and reconsider their budgets and aid priorities; • on top of all this, in 2011 the Programme underwent an internal restructuring to streamline its research development and its research-capacity-strengthening HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 31 processes, and it is once again in the process of handing over its leadership to a new director, who was fortunately appointed, on this occasion, early enough to allow a 3– 4-month period of overlap. Throughout this period, HRP has clearly demonstrated that it is a mature and focused programme. The difficulties it had to bear, it tolerated and adapted to; those that could have changed it for the worse, it resisted; and those that were negotiable, it negotiated in such a way as to maintain its overall purpose and mandate. As will be documented in the following sections of this chapter, the Programme continued to produce many important global public goods in the area of SRH between 2008 and 2012, albeit at a somewhat reduced level due to budgetary constraints. This is largely due to three factors: the dedication and excellence of its staff; the leadership and determination of its directors in making the necessary decisions to ensure that HRP continued to move forward; and its fundamentally sound governance and technical oversight systems, based on its Memorandum of Understanding (see Chapter 1 Annex 2). So, ultimately, because of a robust business model, and an ability to adapt to change, HRP was able to weather the storm and to continue to function very effectively. In the words of one its former directors, “HRP continued to do well, in order to do good”. Continued and increased financial and political support, from donors and cosponsors, will ensure in the future that HRP becomes an even stronger force for change in the world, and an essential component in the assistance countries need in order to attain the Millennium Development Goals (MDGs), and particularly, but not exclusively, the targets for MDGs 4 and 5. 4 The questionnaire In order to gather information and opinions from as wide as possible a variety of persons with knowledge of the Programme, a confidential questionnaire was sent to a group of 416 individuals; 166 persons (39.9%) either fully or partially completed the survey, and a further 10 respondents declined to take the survey for various reasons, but most commonly because of a lack of more recent knowledge of HRP’s work. From here on, this group will be referred to collectively as the “respondents”. The percentages of the different categories of respondents are given in Figure 1. The single largest group of respondents was from the category “member of an HRP technical committee (STAG, GAP, RAP or RP2)”. These numbered 67 and represented a response rate of over 50%. The next largest group was for “current or previous staff member of a cosponsor organization”. There were 20 such responses, representing a response rate of just over 25%. Other respondents were representatives from programme countries (9% of respondents) and donors (6% of respondents). HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 32 Figure 1 Categories of respondents – responses to the question “Please indicate the one category that best describes your relationship to HRP” The length of time respondents had been familiar with the Programme is shown in Figure 2. The majority of respondents, 54%, had been familiar with the Programme for 10 or more years, the majority of these for more than 15 years. A further 22% each made up the categories 0–4 years and 5–9 years. Figure 2 Length of time respondents had been familiar with the work of HRP HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 33 The analysis that follows is based not only on responses to the questionnaire, but also on an extensive review of HRP documentation, interviews with programme staff and additional discussions with other stakeholders and interested parties. 5 Global public health goods Does HRP continue to be a relevant and effective instrument for research in SRH? Does it continue to produce outputs that are consistent with its overall goals and objectives? What has been the outcome/impact of these public goods in programme countries? Global public goods can be defined as those “goods” that are freely available to all, and are non-rival in consumption; that is, consumption by one person does not affect the availability for consumption by others. Global public health goods can be in the form of health knowledge and technologies; health policy and regulatory guidance; and public health systems, including any “good” that makes such systems more effective, more efficient or more accessible. Public health goods created by the Programme include: the results of its research published in peer-reviewed journals; guidelines; policy briefs; programmatic and policy documents; technical guidelines; systematic evidence reviews; global trend analyses in the area of SRH; an electronic journal for the dissemination of evidence and guidance; and new methods of fertility regulation and new technologies in other areas of SRH. For the purpose of this evaluation report, these outputs will from now on be collectively referred to as the “products” of the Programme. An additional global public health good created by the Programme is the many worldwide institutions whose research capacities have been strengthened over the years. This enables these institutions to participate and contribute to the implementation of HRP’s global research agenda, as well as becoming a national public health good, contributing to the implementation of national SRH research agendas. As an example, a number of centres in Africa, including in Zambia and Zimbabwe, are now taking part in the multicentre maternal mortality research project. Some of the more mature centres now play a regional role as a resource centre for the research community in their regions, and as a regional ambassador for HRP, charged with disseminating HRP’s products. 5.1 Priority setting Does HRP continue to use sufficiently robust mechanisms to determine its priorities? Relevance and effectiveness require a sound and systematic approach to priority setting. HRP periodically reviews and assesses its priorities, by engaging groups of experts to assist in identifying, categorizing and ranking research issues. This process, along with the WHO Global reproductive health strategy, adopted at the World Health Assembly in 2004 (1), informs the Programme’s medium-term strategies and biennial budgets and workplans, which are then reviewed by STAG and approved by PCC. The most recent exercise was the development of the 2010–2015 medium-term strategy and involved consultations at both global and departmental levels. At global level, the consultative process involved all regions and identified priorities in the five key thematic areas that form the pillars of the SRH strategy: promoting family planning; improving maternal and perinatal health; preventing unsafe abortion; controlling sexually transmitted and reproductive tract infections; and sexual health, gender, reproductive rights and adolescence. Simultaneously, in collaboration with the Global Forum for Health Research, a consultation was carried out among individuals and organizations, which informed the Medium-Term Strategic Plan (MTSP) 2010–2015. The MTSP was then finalized during two RHR staff retreats. Subsequently, at departmental level, each thematic area of work developed consultative strategies to set more HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 34 operational priorities within the framework set by the Global reproductive health strategy (1) and the MTSP. The Promoting Family Planning team used a process based on the Child Health and Nutrition Research Initiative (CHNRI) priority-setting methodology, in order to identify areas of research for improving the quality of, access to and use of family planning; for new family planning technologies; and for reducing unmet need for family planning. Researchers, representatives of programme countries, technical agencies, donors, service providers and the private sector all contributed to this process. The results were subsequently discussed with stakeholders, who then assigned weighted scores within various areas such as epidemiological research, social science research, health policy and systems research, implementation research and research on contraceptive technology. Occasionally, priorities are triggered by unpredictable events, such as new research findings. A recent example of this was the publication of a study on the relationship between hormonal contraception and HIV transmission, which prompted HRP to convene a technical consultation to review all published research in this area. The meeting resulted in evidence-based guidance on the use of hormonal contraceptives and HIV transmission, as well as recommendations for further research in this area. The Improving Maternal and Perinatal Health (MPH) team organizes a consultation every 4 years, involving major collaborators such as academic institutions, donors, representatives of WHO regional and country offices, and international nongovernmental organizations (NGOs). Participants discuss progress and review a background document, prepared by the MPH team in Geneva, which highlights proposed research initiatives for the future. Space is given for the discussion of additional proposals. These consultations took place in Malaga, Spain in 2008, and in Hua Hin, Thailand in 2012. Maternal and perinatal health normative guideline work is prioritized through a survey of stakeholders, including governments, and WHO and UNFPA regional and country offices. Guideline panels are also used for identifying priority knowledge gaps that require new research. National workshops were conducted in six countries, to determine implementation/scaling-up priorities for evidence-based maternal and perinatal health interventions. The Sexually Transmitted Infection (STI) team developed a log-frame in 2006, through a series of consultations and an STI consultative meeting. This has remained the main instrument that guides the STI workplan. A consultation to review with stakeholders the STI research strategy is planned for early 2013. STI normative work is guided by the STI global strategy (2) and by regional meetings. The Gender and Reproductive Rights team (GRR) receives guidance on priority issues, primarily from GAP, but also from historical reviews and expert meetings, and sometimes from STAG itself. GAP has played a key role in guiding the work of GRR. They call for reviews and prioritization of the work, as was the case with human rights some years ago, and, more recently, GAP specifically requested a review of the work on female genital mutilation (FGM). GAP also asked for prioritization of specific issues, such as gender-biased sex selection and sexual health/sexuality. An expert group meeting subsequently made recommendations on how GRR could best contribute in this area. The Preventing Unsafe Abortion team relies on expert groups with specific expertise on monitoring, guidelines and research. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 35 Proposals for research priorities from each of the five thematic areas are also guided by a log- frame before receiving further guidance from the respective RAPs and STAG. As requested by PCC, these proposals also include three separate levels of priority (V: vital; E: essential; and I: important), based on criteria such as: comparative advantage, public health impact, ongoing implementation and dedicated funding. This enables an even more detailed review according to priorities within each area of work, and is also used to guide the disbursement of programme funds by the secretariat as costs and income fluctuate over time. The priorities of the Programme are also affected by recommendations from its technical oversight mechanisms, STAG and GAP. A review of their recommendations over the last 5 years reveals a more or less continuing pressure on the Programme to do more and add to its agenda, but generally in the absence of either new or additional funds or identification of areas of work that should receive less attention. Only one recommendation in 5 years encouraged the Programme to discontinue work in a specific area. This puts pressure on the Programme to do more with no additional funds, and may thus diffuse rather than focus HRP’s work. Respondents were asked about the influence of programme countries and donors on HRP’s priorities (see Table 1). Table 1 The influence of programme countries on HRP’s priorities Influence Percentage of respondents 1 – very much 7.0 2 44.3 3 28.7 4 12.2 5 – not at all 7.8 Total (n = 115) 100.0 Only 7% of respondents felt that programme countries have a strong voice in determining HRP’s research priorities, and only just over 50% gave a rating of “1” or “2” in answer to this question. When it comes to the influence of donors, 25% of respondents felt that they had too much influence over HRP’s agenda, though 69% felt that the influence of donors was about right. Overall, over 70% of respondents gave a rating of “1” or “2” in answer to the question: “Do the research priorities and programmes supported by HRP address SRH issues most likely to assist programme countries to achieve MDG targets?”. Table 2 shows that less than 20% of respondents considered HRP’s priority-setting mechanisms to be “very strong”, though a further 43% gave this question a rating of “2”. In providing further comments on priority setting, a number of respondents felt that priorities were too donor driven or driven by the interests of individual staff; that STAG should play a stronger role in ranking priorities and carrying out in-depth reviews of specific areas of work; and that HRP’s agenda was still too biomedical and needed to move towards implementation research. Overall, there was an impression that, with the loss of steering committees and specialist panels, the process of priority setting could be strengthened and more focused on a smaller number of critical policy- and programme-relevant questions. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 36 Table 2 Strength of HRP mechanisms for determining its research priorities _Strength Percentage of respondents 1 – very strong 17.9 2 43.4 3 24.5 4 12.3 5 – very weak 1.9 Total (n = 106) 100.0 One question asked of respondents was concerned with missed research opportunities. Almost all respondents replied to this question, indicating a wide variety of subjects, some very specific and some very general. Many informants responded in terms of areas that should receive more emphasis rather than in terms of missed opportunities. Three particular issues were cited more often than any other. These were implementation research, research on adolescents and research on the social determinants of SRH. Respondents also noted research on abortion as a continuing critical area of work for the Programme, and one where its comparative advantage was paramount. Conclusion HRP cannot be expected to research or solve all issues related to SRH. In general, it appears that different thematic areas within the Programme use different methods to identify priorities, and some of these are more rigorous and more inclusive than others. These processes may need to be strengthened to enable the Programme to focus on a smaller number of critical policy- and programme-relevant questions. Respondents gave the general impression that programme countries should have more influence on the setting of HRP’s research priorities and donors a little less, but that, overall, priorities were generally in line with what needed to be done. However, a number noted that it was important for the Programme to keep its ear to the ground and listen and take note of what the real issues were. A number of respondents indicated that the Programme might need to focus more on research that is likely to have an impact in the short term (for example STI prevention, and its work on magnesium sulfate for pre-eclampsia and eclampsia), while maintaining at a more moderate level other areas of work that could have a major impact, but in the much longer term (for example, male contraception). Respondents identified implementation research, research on adolescents and research on the social determinants of SRH as three areas to which the Programme might wish to give greater attention. Recommendations HRP needs to strengthen and take a more uniform approach to its priority-setting process, in order to identify those key research questions and knowledge gaps in SRH that are most likely to have an impact in programme countries. Criteria should include: a priority issue for countries furthest from the MDGs; likely impact; implementability; sustainability; practicality; cost; risk; comparative advantage of HRP; and lead time. In its overall programme of work, HRP should consider giving higher priority to implementation research, research on adolescents and research on the social determinants of SRH. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 37 5.2 Geographical focus Does HRP’s work focus sufficiently on attainment of the MDGs and other global targets, and on the research needs of the least developed countries in overcoming barriers to improving access to SRH information and services? One often-cited barrier to undertaking research in the poorest countries is the lack of both human and institutional resources. However, through its research-capacity-strengthening grants, and its Biostatistics and Data Management unit, HRP has been instrumental in developing methodologies to undertake research in resource-poor settings, supporting the process of design, implementation, monitoring, data management and processing, and publication, while ensuring that good clinical practice (GCP) guidelines, standard operating procedures, and data-quality standards are maintained. Two such recent examples are: • a study on the “Effect of non-pneumatic anti-shock garment (NASG) in the treatment of postpartum hemorrhage”. The study was a cluster randomized trial conducted in 38 rural clinics in Zambia and Zimbabwe, designed to test the effectiveness of the NASG to reduce maternal mortality among women arriving in clinics in hypovolaemic shock due to severe postpartum hemorrhage (PPH). The device acts by channelling blood to the brain, heart and lungs, and has the potential to keep women alive during transportation from lower-level facilities or from the community to referral hospitals; • ”a demonstration project for the implementation of the WHO antenatal care model in Mozambique”. This is a study being conducted in 10 large hospitals across the country that provide antenatal care. It is testing the impact of an intervention to improve the quality of antenatal care by improving the detection, treatment and prevention of major health conditions, such as hypertension, anaemia, HIV/AIDS, malaria, and congenital syphilis, during pregnancy. The first phase of this research identified deficiencies in the supply chain as the main limiting factor for the delivery of antenatal care with an adequate level of quality. An interventions package is now being deployed, which consists of an innovative, sustainable and simple system of commodities, kits and checklists to strengthen the supply chain and empower the nurses that provide antenatal care services. The commonalities between these two projects are that the research questions are highly focused on problems relevant to resource-poor settings, and could not be implemented, or would not provide a relevant answer if undertaken in more developed settings. Robust research designs and methods have been deployed using innovative tools and technologies to assure compliance with GCP, including data quality assurance, patient’s safety and confidentiality, and adequate trial monitoring. The studies were conducted within the health system, and under conditions that reflect the routine delivery of care. In promoting and introducing its products, HRP continues to be open to collaboration with all countries in all regions, on the basis of the past work it has undertaken and future opportunities. In overcoming barriers to improving SRH policies and programmes in countries, the Programme has developed specific methodologies, such as the WHO Strategic Approach, and specific collaborations such as the Strategic Partnership Programme (SPP) with UNFPA. For example, the SPP introduced guidelines that HRP had developed in family planning, maternal and newborn health and STIs/reproductive tract infections (RTIs), using a systematic approach. Within the SPP, a number of countries were defined as “countries of intensive focus”, and these were closely followed up in adaptation and implementation of guidelines after their systematic introduction. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 38 In order to consolidate this process, HRP identified a number of countries for “strategic focus”. The criteria used to select these countries included: high levels of maternal mortality and unmet need for family planning; a broad vision of reproductive health in country programming; commitments made by the government to the United Nations (UN) Global strategy for women's and children's health (3), especially related to family planning; the existence of resources/champions to lead and collaborate joint efforts; countries of intensive focus in the UNFPA/WHO SPP; RHR/HRP's comparative advantage of long-term collaboration; and the potential to scale-up with availability of in-country resources. As a result, six countries from the African Region, and three each from other regions fulfilling the above criteria were selected. These are: • African Region: Benin, the Democratic Republic of Congo, Guinea, Kenya, Nigeria, Zambia; • South-East Asia Region: Bangladesh, Myanmar, Nepal; • Western Pacific Region: Cambodia, the Lao People's Democratic Republic, Viet Nam; • European Region: Republic of Moldova, Tajikistan, Ukraine; • Eastern Mediterranean Region: Afghanistan, Pakistan, Yemen; • Region of the Americas: Peru, the Plurinational State of Bolivia, Guatemala. In comparison, the MDG “countdown to 2015” lists 73 countries with the highest burden of maternal mortality, including the 49 least developed countries, and the H4+1 initiative has identified, as their primary focus, a subgroup of 25 of these countries with the highest rates of maternal mortality. This listing also reflects, to a large degree, those countries with high unmet demand for family planning, poor access to SRH services and high rates of HIV infection. Only 9 of the 21 strategic focus countries of HRP are on the H4+ priority list of 25 countries. However, of the six countries in Africa, five are on the list; of the three in the Eastern Mediterranean Region, two are on the list; and for both the Western Pacific Region and the South-East Asia Region, only one out of the three are on the list. Conclusion It does appear that HRP focuses its work on a set of strategic countries, and has developed methods to undertake research in low-resource settings that ensure quality of research design, implementation and good clinical practice. However, it should align its focus countries more to those targeted by other global initiatives for SRH. Recommendation For HRP to maximize its potential impact, it needs to strengthen its focus on research questions that will benefit the least developed countries and those furthest from the MDG targets, and on undertaking this research wherever possible in these countries. All proposed work should include a clear statement of how it contributes directly or indirectly to the achievement of MDG targets 4, 5 and 6 or any post-2015 global targets. This statement should be used by STAG as a major indicator of the relevance of the proposed research. 1 UNFPA, the United Nations Children’s Fund (UNICEF), WHO, the World Bank and the Joint United Nations Programme on HIV/AIDS (UNAIDS), have joined forces as Health 4+ (H4+) to support countries with the highest rates of maternal and newborn mortality. The H4+ partners support emergency obstetric and neonatal care needs assessments and help to cost national maternal, newborn and child health plans, mobilize resources, increase the number of skilled birth workers, and improve access to reproductive health services. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 39 5.3 Coordination of research Are coordination mechanisms for research both with outside partners and within WHO sufficiently strong? Globally, HRP does not attempt to map the SRH research landscape in any systematic fashion, and neither does any other organization. It is generally agreed that this would be costly and would need to be updated every few years, and that most organizations would not use the information in planning their own research priorities. Its value added would therefore be minimal. The more informal strategy used by the Programme – maintaining a broad overview of the major areas of work being pursued by the global SRH research community through its many formal and informal contacts with them – appears to generally avoid unnecessary and costly duplication of efforts. A good example of this is microbicide research and development, where other organizations are making major investments, and the Programme has decided not to give this priority, instead focusing on the special regulatory and introductory considerations needed for these products. However, in the context of monitoring the implementation of the WHO Global reproductive health strategy (1), which is done by RHR every 2 years, and reported to the World Health Assembly, data are collected to ascertain the extent to which selected research products of HRP are being used or have been integrated in health systems, in countries. Examples include the use of magnesium sulfate, focused antenatal care, emergency contraception, and early detection of cancer of the cervix with acetic acid-aided visual inspection. As can be seen from Table 3, over 80% of respondents gave HRP a rating of “1”, “2” or “3” for the way in which it coordinated its work with research going on in other institutions. Comments by respondents ranged widely, from “excellent” to “a lost opportunity”; there was an overall feeling that more coordination was needed, but that it was now even more difficult because of the greater number of actors in the field and would therefore require more resources. Table 3 Respondents’ views on the effectiveness of HRP in coordinating research with other institutions Effectiveness Percentage of respondents 1 – highly effective 16.1 2 39.3 3 27.7 4 15.2 5 – not at all effective 1.8 Total (n = 112) 100.1a a This percentage over 100% is due to rounding. Within WHO, coordination of research is perhaps most important with TDR and the Department of Maternal, Newborn, Child and Adolescent Health (MCA). It is notable that, towards the end of 2009, there was more formal collaboration among research entities based in WHO, as well as those in Geneva outside WHO. Monthly meetings of these groups shared research in progress, and a key product was the joint publication in PLoS Medicine on defining research to improve health systems, which discussed implementation, health systems and operations research (4). In addition, HRP, TDR and the Alliance for Health Policy and Systems Research have a joint mechanism for coordination of implementation research through the Implementation Research Platform. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 40 TDR is undergoing a major reorientation towards implementation research, and if HRP also decides to move in this direction, then increased and perhaps more formal coordination mechanisms with TDR will be needed. MCA has within its mission statement the following item: “Generate and synthesize evidence and define norms and standards for maternal, newborn, child and adolescent health”. Its work includes basic research, clinical trials and implementation and health-systems research in this area. In HRP, the MPH team has as one of its key objectives: “. developing, assessing and implementing effective interventions” and “… addressing barriers to improving access to quality maternal and perinatal health care …”. Conclusion Given that organizations undertaking research on SRH seem often to have their own agendas, sometimes driven by their donors, a global platform for sharing information may be a sufficient alternative to a more formal attempt at coordination. There would seem to be considerable opportunity for duplication of effort between HRP and MCA, particularly in the areas of maternal and perinatal health and adolescents, which perhaps require stronger efforts in coordinating the planning and implementation of research activities between these two groups. With increasing focus on implementation research, TDR and HRP may need to develop stronger mechanisms for coordination. Recommendations There is a need for a more formal mechanism for coordination of research between HRP and MCA, particularly in the areas on maternal and perinatal research, and research on adolescent SRH; and between HRP and TDR on implementation research. HRP should consider developing an e-platform to enable organizations engaged in research on SRH to share information on their current work and future plans. 5.4 Selected highlights What are some of the major global public goods produced by HRP between 2008 and 2012? What follows is a brief review of HRP’s work in each of its main substantive areas since 2008. (Institution strengthening is not included here, since it is the focus of one of the four individual case-studies undertaken for the evaluation – Chapter 5, Research-capacity strengthening and network building.) In addition, a further case-study examines the Programme’s record on evidence generation and synthesis to improve family planning, prevent unsafe abortion and prevent and control STIs and RTIs (Chapter 4). 5.4.1 Promoting family planning HRP carried out the research and the systematic reviews of clinical and epidemiological evidence needed to develop a guideline entitled Medical eligibility criteria for contraceptive use (MEC) (5). This offers guidance on the safety of using 19 different methods of contraception for women and men with specific characteristics or medical conditions. It is one of HRP’s four cornerstones of family planning guidance, the others being: Selected practice recommendations for contraceptive use (6); Family planning: a global handbook for providers (7); and, a Decision-making tool for family planning clients and providers (DMT) (8). The MEC won the first prize in the Obstetrics and Gynecology category of the 2011 British Medical Association Book Awards. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 41 In order to implement this guidance, the MEC were re-formatted as a simple “wheel” (the MEC wheel) (9), in association with the INFO project at Johns Hopkins University, the Partnership for Communication for Family Health in Jordan, and the University of Ghana Medical School. The MEC wheel converts the recommendations into a single hand-held tool that can be used directly by service providers when counseling clients on choice of contraceptive method. The MEC wheel has already been adapted, translated into 24 languages and printed and distributed in all regions. There is evidence of its use in over 80 countries, and this is clearly a minimum figure, since it is not based on a universal survey. The information comes from WHO sales figures (over 135 000 copies in 38 countries); replies from respondents (28 additional countries), and the UNFPA survey (14 additional countries). The MEC wheel is also available in a computer-based electronic version for training or demonstration, and as a mobile phone app, which allows family planning providers even greater access than through traditional paper or electronic formats. The MEC are subject to continuous updating through a systematic mechanism known as CIRE (Continuous Identification of Research Evidence). A weekly review of the family planning literature, carried out jointly by HRP and the Centers for Disease Control and Prevention (CDC) in Atlanta, Georgia, United States of America (USA) enables early identification of any new evidence or issues that would affect the recommendations. HRP’s work in the area of family planning also includes initiatives in the area of infertility. These range from research to simplify clinic methods for managing infertility in low-income settings, to regular updating of the WHO laboratory manual for the examination and processing of human semen (10). This publication is a laboratory guide used, for example, in the clinical evaluation of infertility. The manual, released in its fifth edition in 2010, draws on the expertise of a wide group of HRP’s collaborators in the field of male reproductive health, and is the document most frequently downloaded from the WHO web site. It is already available in six languages (Chinese, English, German, Italian, Japanese and Turkish), and additional translations are under way. Over 4500 copies of the fifth edition have already been sold, and more than 500 copies distributed free of charge in programme countries. As of the end of 2012, downloads from the HRP web site number 27 500. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 42 Boxes 1 and 2 present utilization headlines for these publications. Box 1. Utilization headlines for MEC, the MEC wheel and the decision-making tool In Mexico, the Ministry of Health disseminated 10 000 copies of the MEC wheel (9) and evaluated its use in clinical settings. Preliminary results from the sample of 161 doctors interviewed found that 80% said they had used the wheel in the last 2 weeks and, among users, 84% found it “easy or very easy” to use, and 91% found it “useful or very useful” in their work. In the Americas, in collaboration with the Centro Latinoamericano de Perinatologia/Salud de la Mujer y Reproductiva, 5000 Spanish language copies of the DMT (8) and Family planning: a global handbook for providers (7) were distributed among 18 countries. As a result, a number of countries, including Cuba, Guatemala and the Plurinational State of Bolivia, either updated existing or implemented new family planning service norms. In Argentina, Brazil, Paraguay, Peru and Venezuela, the MEC (5) and the MEC wheel (9) are being reprinted and used. In Argentina and Venezuela, the governments are underwriting the costs, and in other countries support is being provided by UNFPA. The DMT (8) and Family planning: a global handbook for providers (7) were used as the basic documents in a workshop for 15 English- and Dutch-speaking Caribbean countries. Participants, who came from ministries of health and NGOs providing family planning services, developed draft workplans for the use of these guides in their national programmes. Chile has used HRP products in the development of its national norms on fertility regulation, for the provision and use of contraceptives in the national health system. The MEC (5) have been used to develop national family planning service-delivery standards, and preservice and in- service family planning training for all levels of providers in the United Republic of Tanzania. In the Philippines, the UNFPA country office financed the adaptation, translation and printing of 40 000 copies of the MEC wheel (9). In a number of countries in Asia, notably Bangladesh, China, Indonesia, Mongolia, Myanmar, Solomon Islands, Tonga, Vanuatu and Viet Nam, HRP’s family planning guidance tools have been used to formulate/update reproductive health policies, update standards and national guidelines, improve training curricula, conduct training activities, develop advocacy materials and promote service-delivery improvements. Many of these came about as a result of the WHO/UNFPA SPP, which, until 2008, provided HRP with additional funds for the introduction, adaptation and dissemination of its evidence-based guidelines in countries. The Mongolian version of the DMT has been widely distributed throughout the country and is regularly used in family planning clinics. Myanmar simplified the DMT (8), which is now used by community health workers. In Nepal, the Ministry of Health has expanded the use of the DMT to 40 villages in four districts. The DMT (8) has been translated into Lao and is being used in orientation workshops for service providers. The MEC (5) have been used to revise clinical family planning protocols in the States of Uttar Pradesh and Bihar in India. Sri Lanka has adapted the MEC wheel (9) for use in its national programme. The MEC (5) and the DMT are both used in Iran to improve the quality of family planning information and care. The MEC (5) have been used to develop and/or revise national guidelines in Latvia, Romania, Turkmenistan and Uzbekistan. Armenia, Azerbaijan, Kyrgyzstan, Latvia, Lithuania, the Russian Federation and Serbia, among several other countries in the European Region, have translated and printed the MEC wheel (9), and then used it for improving the quality of family planning services at primary health care. The MEC (5) is used in many high-income countries, such as the United Kingdom of Great Britain and Northern Ireland (UK) and USA, to inform national guidelines and decisions made by drug regulatory authorities. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 43 Box 2. Utilization headlines for other HRP family planning products As a result of HRP’s research, Afghanistan changed its policy to enable trained nurses and midwives to insert intrauterine devices, thus greatly expanding the availability of this method. The Sichuan Family Planning Research Institute in Chengdu, China, which received long-term institutional strengthening grants from HRP, established non-scalpel vasectomy in China and is now recognized as the national reference centre for training and dissemination, and as an international training centre. Using evidence-based family planning guides and tools developed by HRP, a master training of family planning trainers was carried out in Iraq with collaboration from the Ministry of Health in Jordan. In developing national guidelines and standards, Turkey uses the guidance and recommendations from WHO/HRP’s publications on family planning, and, more generally, uses HRP’s products as a reference point on all other SRH interventions. The International Planned Parenthood Federation (IPPF) works with its member affiliates in all countries to promote the adaption and adoption of HRP guidance on family planning, as well as broader guidance on all SRH issues. Member affiliates extend and expand this process by working with the national authorities to promote the use of HRP’s products. A survey of programme countries in 2011 revealed that around two thirds of those responding had included emergency contraception as one of the family planning methods available in public programmes. Over 80% of respondents felt that the MEC (5) had been used to strengthen SRH policies and/or programmes. Conclusion HRP’s MEC (5), the MEC wheel (9) and the DMT (8) are being widely used in many programme countries throughout the world, and are undoubtedly having a positive impact on quality of the SRH of individual women, men and adolescents. Other HRP products are also expanding the availability of family planning technologies. Recommendation To further strengthen its effectiveness, the Programme should examine if and how the CIRE approach could be used for the other thematic areas of HRP’s work. 5.4.2 Maternal and perinatal health In the area of maternal and perinatal health, HRP has addressed some key questions of high relevance between 2008 and 2012, and as a result has produced a number of global public health goods, many of which are already being used in programme countries. Management of the third stage of labour HRP undertook research on the management of the third stage of labour, which demonstrated that in situations where injectable uterotonics (primarily oxytocin) were available, the use of misoprostol did not have any additional benefits in preventing PPH. Additional research in this area showed that controlled cord traction was safe in hospital settings but could be omitted, if injection of oxytocin was available, in situations where skilled birth attendants were not present. After publication of the results in peer-reviewed journals, the WHO guideline on PPH treatment and management has been updated and is in press, and further dissemination of the guidance has started. A meeting cosponsored by the United States Agency for International Development (USAID) and the Maternal and Child Health Integrated Project in Dhaka, Bangladesh, with over 300 participants from various programmatic backgrounds, presented the results and their implications for national programmes. USAID is also issuing a technical note to all its country offices and contract agents, to change their approach on management of the third stage of labour. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 44 The recommendations on misoprostol were subsequently incorporated into the 17th edition of the WHO Model list of essential medicines (11), a reference guide widely used by programme countries in determining national drug formularies. Optimizing maternal and newborn care HRP undertook the research and analysis that enabled the development of evidence-based recommendations to facilitate universal access to key, effective maternal and newborn interventions through optimizing the roles of health-care workers. These recommendations are intended for health policy-makers, managers and other stakeholders at regional, national and international levels and can be adapted to specific national contexts. The guideline: WHO Recommendations for optimizing the delivery of key maternal and newborn health interventions (in press) makes a total of 128 recommendations, 25 of which are for lay health workers, 27 for auxiliary nurses, 22 for auxiliary nurse-midwives, 17 for nurses, 13 for midwives, 8 for associate clinicians, 8 for advanced associate clinicians and 1 for non-specialist doctors. Two launch meetings have already taken place in 2012, the first with WHO Regional Office for Africa, in Addis Ababa, and the second at the International Federation of Gynecology and Obstetrics conference. The Reproductive Health Supplies Coalition is considering the production of a policy brief, and supporting further dissemination. This guideline provides the scientific basis for various task-sharing and task-expansion activities by different categories of health-care workers in different clinical settings. It will ultimately result in greater access to appropriate and safe maternal and newborn care for women. Maternal “near-miss” HRP developed a standardized definition of maternal “near-miss” (12), and created criteria for its identification, in order to improve monitoring of maternal care. This allows assessments of the management and response to complications occurring during pregnancy and the intrapartum period, and thus serves as a barometer of the maternal care system. By standardizing the definition, the robustness of the evidence base is improved, and programme managers are better able to monitor and assess performance in health-care facilities. The Programme also undertook a multicountry survey on maternal and newborn health (13). This study, conducted in over 350 health facilities in 29 countries aims to evaluate the quality of care in health facilities, through examination of the prevalence of maternal “near-miss” cases. The findings of the study will enable a more comprehensive dialogue with policy- makers, programme managers, professional associations and civil society, to promote best practices, improve quality of care and achieve better health for mothers and children. A facility-level monitoring tool with indicators has already been published. HRP carried out the research showing the effectiveness of magnesium sulfate in the treatment of pre-eclampsia and eclampsia, and subsequently ensured that it was included in the WHO Model list of essential medicines (11). Other important evidence-based guidelines on maternal and perinatal health include: WHO Recommendations for induction of labour (14); WHO Recommendations for prevention and treatment of pre-eclampsia and eclampsia (15); and Managing complications in pregnancy and childbirth (16). All of these, as well as all other HRP guidelines and systematic reviews, are available through the Reproductive Health Library (RHL) both online and as a CD-ROM. Finally, HRP has initiated collaboration with the inventor of the Odon Device, a low-cost, easy- to-use technological innovation to facilitate operative vaginal delivery and minimize trauma to HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 45 the mother and baby. The device has great potential for use by mid-level health-care providers in low-resource settings, and is currently in early clinical trials in Argentina and South Africa. Box 3 presents utilization headlines for maternal and perinatal care products. Box 3. Utilization headlines for maternal and perinatal care products In order to have a criterion-based clinical audit method for improving the quality of maternal care, the HRP near-miss monitoring tool is being used in Peru as a national policy for the surveillance of severe maternal morbidity. Using the results of HRP’s research, national-level interventions for the improvement of maternal, newborn and child health were introduced in Brazil, Chile and Mongolia. Maternal “near-miss” has been adopted for use in Uganda. Using evidence-based guidelines developed by HRP, Myanmar has adopted a new national policy on active management of the third stage of labour. Definition of maternal death, and maternal death audits have contributed to improving maternal care in Mozambique, and in training health-care providers in Pakistan. In Mozambique, during the implementation of a cluster randomized trial, it became apparent that shortages and rupture of stocks of commodities prevented implementation of the model, and this led the Ministry of Health to develop an intervention to address commodity supply issues. Fifteen other countries are implementing the antenatal care model developed by HRP, which halves the expense and the time women spend in accessing services, without compromising the quality of care received. A survey of programme countries in 2011 revealed that 95% of those responding had registered magnesium sulfate in their national lists of essential medicines. As can be seen from Table 4, almost 90% of respondents cited the global estimates as having an impact on programmes and policies. For the other three products, over 60% gave a positive response, with a much higher proportion of “Don’t knows”, most likely because these products are relatively new in the HRP “pipeline”. Table 4 Respondents’ views on the use of maternal and perinatal care products to strengthen sexual and reproductive health programmes or policies Product Percentage of responses Yes No Don’t know Total Global maternal mortality estimates (17) 88.9 2.2 8.9 100.0 Maternal near-miss definition/criteria (12) 63.4 3.0 33.6 100.0 Recommendations on maternal and newborn health interventions (in press) 65.4 3.8 30.8 100.0 Multicountry survey on maternal and newborn health (13) 60.7 5.2 34.1 100.0 5.4.3 Preventing unsafe abortion Every year, approximately 22 million unsafe abortions take place globally, and 98% of these occur in low- and middle-income countries. The global rate of unsafe abortion has remained somewhat unchanged since 2003, though the absolute numbers have increased by 2 million between 2003 and 2008. Complications of unsafe abortion are estimated to result annually in 47 000 deaths, and an additional 5 million women suffer from a subsequent disability. Globally, 1 in 8 maternal deaths are attributable to abortion, and in Eastern Africa this rises to 1 in 5. Technical and policy guidance on safe abortion HRP updated and published in 2012, Safe abortion: technical and policy guidance for health systems (18), a document that provides a set of the latest evidence-based recommendations. The document was developed with inputs from various groups of experts as well as HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 46 information from Cochrane systematic reviews. All available evidence was appraised and graded using a standardized approach. The evidence and recommendations then went through a final review at a technical consultation at WHO. The document is an essential tool for support to countries, including strategic assessments on unintended pregnancy and abortion; development and revision of national standards and guidelines for comprehensive abortion care; and technical opinions on restrictive laws and parliamentary amendments. A review and update will take place in 4 years. In parallel, a companion document, Clinical practice handbook for safe abortion care is being developed by the Programme. Mid-level providers for safe abortion Randomized controlled equivalence trials conducted by HRP showed that trained mid-level health-care providers can administer early surgical abortion (South Africa and Viet Nam) and medical abortion (Nepal) as safely and effectively as doctors. These findings have a major impact on preventing unsafe abortion and related morbidity and mortality. In countries where abortion is not against the law, it potentially expands access to safe abortion through mid-level providers, especially in remote areas where doctors are in short supply, and for poor women who are unable to afford doctors’ fees. Box 4 presents utilization headlines on unsafe abortion. Box 4. Utilization headlines on unsafe abortion Between 2008 and 2011, HRP collaborated with eight countries that wished to better address the issue of unintended pregnancy and abortion. Using the “strategic approach”, a strategic planning and policy and programme implementation tool previously developed by HRP, which includes assessment, introductory and scaling-up phases, these eight countries have now made considerable progress towards the goals they themselves set. Four countries in Africa identified similar barriers to safe abortion and pregnancy prevention. These included poor implementation of existing laws, which limited women’s access; lack of knowledge about the legal status of abortion by both clients and providers; costly, clandestine systems for abortion; inadequate post-abortion care; weak family planning information, education and service systems; and religious, social and cultural barriers. A recent follow-up review concluded that the commitment and actions of governments and civil society stakeholders with regard to reducing abortion-related deaths and injuries had significantly increased. In the Republic of Moldova and Ukraine, HRP’s strategic approach enabled: the development of national standards and guidelines on comprehensive abortion care; revision of preservice training curricula based on the national guidelines; and the development of model services. In Kyrgyzstan, the strategic approach led to development and approval of a new national clinical protocol on comprehensive care for unwanted pregnancies; the introduction of a client-oriented approach in clinics; and, the development, testing, approval and implementation of new training curricula. Using HRP guidelines, the former Yugoslav Republic of Macedonia developed national norms and standards for abortion care in response to a previously undertaken strategic assessment exercise. HRP safe abortion guidance has been used to improve the quality of abortion care in Romania and the Russian Federation. In 2012, Bangladesh adapted the HRP document Safe abortion: technical and policy guidance for health systems (18) as the basis for updating their national menstrual regulation services guidelines. The Programme provided technical assistance to Nepal for scaling up the introduction of methods of medical abortion developed by HRP. Based on the original study carried out in Nepal, South Africa and Viet Nam, an ongoing operations research project is assisting 30 countries to expand access to medical abortion in low-resource settings, again using the strategic assessment approach. The Ministry of Health in Colombia used HRP guidance documents to develop national guidelines on abortion. The Ministry of Health in Argentina is training health-care professionals using the project team that undertook the research related to improving post-abortion care, including use of family planning. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 47 A total of 82.2% of respondents felt that safe abortion guidance had strengthened SRH programmes or policies, and two out of every three felt that research on using mid-level providers for carrying out safe abortion had had a similar outcome. Conclusion This work clearly demonstrates how HRP develops and uses products to facilitate outcomes in programme countries, such as improved standards of care, which have an impact on the health and lives of women. This is also a clear example of the comparative advantage of HRP – its ability to work in sensitive areas with the complete confidence of countries. 5.4.4 Prevention and control of sexually transmitted and reproductive tract infections including gynecological cancers and infertility Sexually transmitted and other reproductive tract infections The HRP publication Sexually transmitted and other reproductive tract infections: a guide to essential practice (19) is a reference manual and a resource to educate and remind health-care workers of the need to consider STIs and RTIs when providing other SRH services. It is used widely for preservice and in-service training; as a source of up-to-date, evidence-based recommendations; and as a starting point for improving policies and programmes for the prevention and management of STIs/RTIs. Rapid test for syphilis Syphilis screening in pregnancy is more effective in preventing stillbirths than any other pregnancy intervention besides comprehensive emergency obstetric care, and costs less per pregnant woman than nearly any other intervention. HRP, in collaboration with TDR, developed a point-of-care rapid test for syphilis that can be administered by any trained antenatal care provider. The test provides an immediate result and thus the possibility of immediate treatment. Evidence has shown that introduction of the rapid test can increase the coverage of syphilis screening in antenatal settings. Research has also included field-studies to better understand implementation, acceptability and scaling-up issues. The Programme is currently developing an advocacy tool for investing in this intervention: Investment case for eliminating congenital syphilis: promoting better maternal and child health and stronger health systems. Preventing cervical cancer HRP has assessed the acceptability and feasibility of implementing a “see and treat” approach to cervical cancer, using visual inspection with acetic acid and cryotherapy. With some basic training and the addition of some minimal equipment and supplies, the test can be administered by nurses and midwives and the treatment can be provided at a single visit. These services are now being scaled up in the six countries where the research was originally conducted, and five of the six have already developed national plans. Other guidelines produced include: technical guidelines on Strategies and laboratory methods for strengthening surveillance of sexually transmitted infections (20); and on Use of cryotherapy for cervical intraepithelial neoplasia (21). HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 48 Box 5 presents utilization headlines on STI/RTI products. Box 5. Utilization headlines for STI/RTI products In four Central Asian countries (Kyrgyzstan, Tajikistan, Turkmenistan and Uzbekistan) and in partnership with UNFPA country offices, guidelines developed by HRP were used for the integration of family planning and management of STIs at primary care level. The Programme provided support to Rwanda, the United Republic of Tanzania and Zambia to strengthen their national programmes for cervical cancer prevention, including the introduction of human papillomavirus (HPV) vaccine. Using the results of its research, the Programme has ensured the updating of the WHO essential medicines list (11), to include the relevant antibiotics for resistant STIs. A total of 77.6% of respondents felt that the HRP manual Sexually transmitted infections and other reproductive tract infections: a guide to practice (19) had been used to strengthen SRH programmes or policies in countries. 5.4.5 Sexual health including adolescence, gender and sexual and reproductive rights A standardized methodology for measuring violence against women HRP contributed to the development of a standardized methodology to assess violence against women, particularly intimate partner violence (physical, sexual, emotional and controlling behaviours) and its health consequences, as well as risk and protective factors. Standardizing such measurement has enabled comparative studies across and between countries, and facilitated the use of such data for policy-making and programming. This training programme aims to promote the integration of gender equality and human rights in reproductive health policy, programmes and research. It has been translated into multiple languages and adapted in different ways. Interagency statement on preventing sex selection The Office of the High Commissioner for Human Rights (OHCHR)/UNFPA/UNICEF/UN Women/WHO statement, Preventing gender-biased sex selection (22) highlights the public health and human rights dimensions and implications of the problem and provides recommendations on how best to take effective action. Box 6 presents utilization headlines for HRP gender and adolescent products. Box 6. Utilization headlines for HRP gender and adolescent products The Global strategy to stop health care providers from performing female genital mutilation (23) is credited with strongly influencing a Kenyan Law against FGM, passed in Parliament in October 2011. The Ministry of Health in Senegal used the results of HRP’s research to develop its first action plan for improving adolescent SRH services. In Panama, the national education policy changed as a result of research undertaken by an HRP collaborating centre. Teachers are now trained to discuss issues of sexuality with students, and pregnant schoolgirls are allowed to continue their studies. Research in Shanghai, China found that young people considered a dedicated web site to be the best way of improving their knowledge about SRH. The web site is now part of the “life education” programme for secondary school students. The tool for examining laws, regulations and policies in relation to SRH and human rights (unpublished) and its adaptation for adolescent SRH was applied in Moldova, Sri Lanka and Tajikistan. Gender training carried out in China, Myanmar and Sudan led ministries of health to examine how to integrate gender and human rights issues into reproductive health policies. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 49 A total of 62.1% of respondents felt that the HRP guideline Gender and rights in reproductive health: a training manual for health managers (24) had strengthened SRH programmes or policies in countries. 5.4.6 The UNFPA country office enquiry In order to assess more objectively and quantitatively both the use of the Programme’s products in countries, and its modes of communication, a short questionnaire was sent to all UNFPA country offices. The rationale for choosing UNFPA was that it is the HRP cosponsor, whose substantive mandate corresponds mostly closely with that of the Programme. Thirty- four replies were received by the deadline, a response rate of just over 30%. The first question enquired about the use of the Programme’s products to strengthen SRH policies and/or programmes in their countries. Table 5 indicates the proportion of respondents who gave a positive response for each of the products listed. Table 5 Proportion of UNFPA country offices that confirmed that the product had been used in their country Product Positive responses (%) “Don’t know” (%) MEC (5) 88.2 12.1 The MEC wheel (9) 55.9 33.3 Kesho Bora study results (25) 8.8 60.6 Maternal mortality estimates (17) 82.4 3.0 Maternal near-miss definition (12) 55.9 30.3 Safe abortion guidance (18) 61.8 18.2 Research on mid-level providers for abortion 20.6 36.4 Optimizing delivery of maternal and newborn health interventions (27) 67.6 27.3 Multicountry survey of maternal and newborn health (13) 35.3 51.5 RHL 55.9 27.3 STI guide to essential practice (19) 76.5 15.2 Gender and rights training manual (24) 52.9 27.3 More than two thirds of respondents gave a positive response to four products – MEC (5); maternal mortality estimates (17); Optimizing delivery of maternal and newborn health interventions (28); and the STI guide to essential practice (19). Over 50% responded positively on 9 of the 12 products. For those products with low ratings, “Don’t know” responses were correspondingly high, Kesho Bora (60.6%) (25); research on mid-level providers for abortion (36.4%); and, the multicountry study on maternal and newborn health (51.5%) (13). Respondents elaborated their responses by giving many instances of where these products had been used to inform the development of national policies and strategies; to guide the development of national norms, standards and clinical protocols; to update training curricula; to strengthen training programmes; to improve the quality of service delivery including counselling; and to launch national initiatives such as on reducing maternal mortality. Two particularly detailed responses from Nigeria and Tajikistan are provided in Boxes 7 and 8. The second question enquired about the effectiveness of HRP in providing guidance on norms and standards. Over 80% of respondents gave a rating of “1”, “2” or “3” for the effectiveness of the Programme’s work in this area. The final questions in the UNFPA survey related to modes of communication, and suggestions for their improvement, and these findings are reported in Section 10 of this chapter, on the efficiency and effectiveness of HRP’s communication. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 50 Box 7. UNFPA Nigeria response Nigeria used the MEC (5) in its review of the national family planning protocol, and the MEC wheel (9) is used by family planning providers. HRP’s products in the area of maternal and neonatal health helped focus greater national attention on maternal mortality, and were used in the development of the national maternal, neonatal and child health strategy, 2012–2017 (27). HRP’s documents, including the one on safe abortion (18) have been used in the development of the country’s reproductive health policy, and for the training of mid-level health workers. The national STI training manual was guided by HRP’s guide to essential practice for STIs (19). The gender training manual (24) is used in the training of health-care providers on how to mainstream gender in the provision and management of reproductive health services. The RHL is a continuing source of information for updating training manuals, and was also used as a resource in the development of the national reproductive health policy. Box 8. UNFPA Tajikistan response The UNFPA country office assisted various capacity-building initiatives for the MEC (5), including training of trainers, and subsequent training of service providers. The MEC wheel (9) was printed and disseminated widely throughout the country. Between 2008 and 2012, the UNFPA country office in Tajikistan conducted a number of national capacity-building activities to implement evidence-based strategies to optimize delivery of key maternal and newborn health interventions, including “Beyond the numbers” (BTN), an audit of maternal near-miss cases, and “Effective perinatal care” (EPC). As a result of these interventions, the coverage of maternal and neonatal care was increased, and the quality of service was improved. BTN started in 2008 with a confidential enquiry into maternal deaths and a “near-miss” analysis, and subsequently a strategic partnership was established between WHO, USAID and the German Development Agency (GTZ), to implement the findings. EPC introduced a set of effective perinatal care technologies through updating training curricula, training of trainers workshops, cascade training, and follow-up monitoring and mentoring. As a result, there was a decrease in the number of deliveries with complications; early neonatal mortality and morbidity decreased; skilled birth attendance increased; facility-based deliveries increased; and bleeding, postpartum infection and haemorrhagic shock declined Using HRP guidelines, the UNFPA country office was also able to provide input to the development of the National Strategy on Safe Abortion, recently approved by the government. National standards on safe abortion were developed and are being implemented. Using HRP guidance on STIs, training modules on STIs were updated and used for basic and in-service training. The Gender training manual (24) was used to conduct a number of training sessions for health managers, and subsequently UNFPA supported the establishment of safe rooms in health-care facilities for victims of gender-based violence (GBV). Conclusion This far from comprehensive review of just a sample of HRP’s products revealed evidence of their use in over 130 programme countries. The UNFPA country office enquiry found that four critical products – MEC (5), maternal mortality estimates (17), optimizing delivery of maternal and newborn health interventions (28), and the STI guide to essential practice (19) – had been used in more than two out of three countries. A more comprehensive survey would surely find many additional examples of country use. It would appear that many countries are making very good use of HRP’s products in strengthening their SRH programmes, but that even more could be benefiting. This will require investment in a new communication and uptake strategy for all the Programme’s products, and investment in an introductory strategy to be able to demonstrate in a limited number of countries how some of HRP’s products can be incorporated into SRH programmes (see Section 10). These “success story” examples can then be used to leverage the larger funds of multilateral and bilateral donors, including the cosponsors and foundations, for use in additional countries for similar purposes. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 51 5.4.7 Overall conclusions and recommendations HRP continues to ensure its relevance by being the unique global resource that generates the research findings, synthesizes the evidence and develops the products to support policy formulation and programme strengthening to improve SRH. HRP’s outputs continue to be consistent with its overall goals and objectives. It continues to provide global leadership on sensitive SRH issues, and it continues to generate global public health goods of the highest quality and utility. Recommendation The Programme should commission a periodic review of the utilization of its products in programme countries, and estimates of their potential or actual impact. Such a review should also identify the source of funds, whether from national governments, cosponsors, other international donors, bilateral donors, foundations, NGOs, etc., that were used to support the introduction. Such a review will demonstrate the value of investing in HRP and thus further strengthen its fundraising ability. 6 Comparative advantage Does HRP have a comparative advantage and does it continue to utilize it? WHO provides a forum that is unique in many ways. It is universally owned by countries; it is inclusive and neutral; it has the ability to work on the widest spectrum of health issues however sensitive, and to convene health authorities and experts to deliberate on health topics; and when WHO speaks, its “imprimatur” or seal of approval is universally recognized. HRP benefits from this mantle and adds to it a commitment to the highest quality of research, science and evidence. Some examples of HRP’s comparative advantage include its work on preventing unsafe abortion, and on adolescent SRH, and violence against women. HRP’s work on unsafe abortion includes methods for non-surgical termination of pregnancy; monitoring global trends on unsafe abortion; advocacy and programme research to increase the use of family planning methods in countries where abortion has traditionally been used as a method of regulating fertility; research on mid-level health personnel as providers of safe abortion services; and dosage studies of mifepristone and misoprostol by vaginal or sublingual administration. In the area of adolescent SRH, HRP’s research continues to legitimize work in this area, and to demonstrate the need to deal openly with the issue. Countries that have collaborated with the Programme in the area of adolescent health have strengthened their capacity to confront and address the issues of SRH among young people. In the area of sexuality education, the Programme has supported the development of models and materials for sexuality education, which can be adapted for use in countries. In the area of violence against women, the Programme has been instrumental in developing consensus on a number of issues, for example, the elimination of FGM (29); the medicalization of FGM (30); violence against women (31); and preventing gender-biased sex selection (32). HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 52 6.1 Respondents’ views on HRP’s comparative advantage On a scale of 1 to 5 (1: highly effective to 5: highly ineffective), the following proportions of respondents gave HRP’s comparative advantage characteristics a rating of “1” or “2”: • neutrality – 78.3%; • convening power – 78.2%; • WHO seal of approval – 84.0%; • ability to address any SRH issue however sensitive – 80.8%; • commitment to the highest standards of research – 76.0%. These figures clearly confirm the confidence respondents have in the comparative advantage of HRP. In its decisions on proposed research, STAG is guided by a number of principles, including quality of proposed research, likely impact, and areas of research neglected by other organizations engaged in SRH research. Application of this last principle additionally strengthens the comparative advantage of HRP. Respondents were also asked if other organizations existed that could fulfil a similar function to HRP. Over 87% of respondents indicated that there was no other such organization. Responses to the question concerning HRP’s convening ability are given in Table 6. Table 6 Effectiveness of HRP in convening groups of experts Effectiveness Percentage of respondents 1 – highly effective 45.3 2 33.3 3 14.5 4 4.3 5 – not at all effective 2.6 Total (n = 117) 100.0 As can be seen from Table 6, an overwhelming proportion of respondents (over 78%) gave a rating of “1” or “2” for HRP’s convening ability. Respondents also felt that HRP could be doing a number of things more or better to use its comparative advantage. These included: strengthening the communication and marketing of its products to programme countries, directly and through international and regional partners, to better translate knowledge and evidence into action; carrying out more implementation research; and strengthening collaboration with research institutions. Overall, respondents felt that preventing unsafe abortion was one very specific area where HRP had a major comparative advantage over other organizations. Conclusion The Programme continues to demonstrate its comparative advantage through its ground- breaking work in areas such as unsafe abortion, adolescent SRH and violence against women. It continues to exploit its neutrality, its inclusiveness and its ability to convene the broadest array of interested parties to discuss and provide guidance on sensitive technical and policy issues in the area of SRH. The value of its guidance and other products is maximized by its position within WHO. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 53 7 Norms and standards Does HRP continue to set policy and programme norms and standards? HRP’s work in the area of norms and standards results in various types of publication, including: technical (clinical) guidelines, programmatic and policy documents and policy briefs. These set global standards for policies, programmes and clinical practice in SRH. In the past 5 years the Programme has produced 86 such documents, including 16 clinical guides, 51 programmatic and policy documents and 19 policy briefs (see Box 9). Box 9. Headline examples of the Programme’s work on norms and standards over the last 5 years • The four cornerstones of family planning, including the MEC (5) • WHO/HRP statement on hormonal contraception and the risk of HIV infection (33) • Fifth edition of the WHO laboratory manual for the examination and processing of human semen (10) • WHO recommendations for prevention and treatment of pre-eclampsia and eclampsia (15) • WHO recommendations for induction of labour (14) • A framework for the standardized classification and quantification of causes of maternal mortality – contributing; underlying; and immediate (34) • WHO guidelines for the management of postpartum haemorrhage and retained placenta (35) • Safe abortion: technical and policy guidance for health systems (18) • Sexually transmitted and other reproductive tract infections: a guide to essential practice (19) • Global reference for fetal- and birth-weight percentiles (36) • The OHCHR, UNAIDS, UNDP, United Nations Economic Commission for Africa (UNECA), United Nations Education, Scientific and Cultural Organization (UNESCO), UNFPA, United Nations High Commisisoner for Refugees (UNHCR), UNICEF, United Nations Development Fund for Women (UNIFEM), WHO interagency statement on Eliminating female genital mutilation (29) • The OHCHR, UNFPA, UNICEF, UN Women, and WHO interagency statement on Preventing gender-biased sex selection (22). In addition HRP, in collaboration with its partners, standardizes SRH terminology, contributes to relevant sections of the WHO model list of essential medicines (11), ensures that essential reproductive health commodities are added to the WHO Essential Medicines Prequalification Scheme, provides global reference standards in the area of SRH, updates relevant sections of the International statistical classification of diseases and related health problems (37), specifically Chapters 14–16 on SRH, including developing standard definitions of items such as a skilled birth attendant, and how to interpret this definition in different country settings. For example, HRP’s input to the WHO model list of essential medicines (11) includes the following: • the use of misoprostol was expanded and clarified, particularly its role in the management of incomplete abortion and miscarriage, in the 16th edition; • in the next edition, issued in 2011 (11), the use of misoprostol was further clarified with regard to PPH and induction of labour, again on the basis of research undertaken by the Programme; • also as a result of an HRP review of evidence and advice from a technical consultation, ethinylestradiol was determined not to be an essential medicine and was removed from the model list in the latest (2011) edition (11). HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 54 Respondents’ views of HRP’s work on norms and standards are presented in Table 7. Table 7 Effectivenes of HRP’s work on norms and standards Effectiveness Percentage of respondents 1 – highly effective 33.1 2 42.1 3 14.9 4 5.8 5 – not at all effective 4.1 Total (n = 121) 100.0 As can be seen from Table 7, overall, on a scale of 1 to 5, where 1 is highly effective, more than 75% of respondents gave a rating of “1” or “2” for the Programme’s work on norms and standards. Many respondents felt that the work undertaken by HRP on the development of evidence- based norms and standards was critical, and an area that was uniquely appropriate for WHO. Respondents also felt that the process of developing guidance was occasionally somewhat cumbersome, and needed to be simplified and expedited whenever possible, without losing any of its rigour. A total of 52.9% of respondents felt that HRP should do more work on norms and standards, while a further 35.3% felt that the proportion of this work was “about right”. A total of 87.2% of respondents felt that the maternal mortality estimates had been used to strengthen SRH policies or programmes in countries, and 67.6% responded similarly on HRP’s work on the criteria/definition of maternal death near-miss. Conclusion HRP continues to be the gold standard for developing, monitoring and updating the evidence- based norms and standards required to guide SRH policies, strategies, programmes and clinical practice. Policy statements, programme guides, clinical guidance, and evidence summaries issued by WHO/HRP are key reference materials for governments when developing or revising SRH policies and programmes. As an entity within WHO, the credibility of HRP among Member States is assured, and its materials thus receive far greater attention, and have a larger global health impact than similar issuances by any other institution. 8 Monitoring of global trends in sexual and reproductive health Does HRP continue to monitor important global trends in SRH? A few examples of HRP’s important work in the area of global trends follow. 8.1 Global maternal mortality estimates. HRP, in collaboration with the WHO Health Information and Statistics Department, UNICEF, UNFPA and The World Bank analyses maternal mortality levels on a routine and continuing basis. The updated maternal mortality estimates (17) allow for trend analysis to determine the progress towards attaining MDG 5a. The methodology and process undertaken for these estimates is intended to ensure both international comparability and ownership by countries, and provides a starting point for building capacity in-country on improved systems for data collection and analysis. The estimates also include an analysis of regional trends. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 55 8.2 Global estimates of unsafe abortion In 2011, HRP published the sixth edition of the global estimates of unsafe abortion and associated mortality (38), a document aimed at policy-makers and programme managers, to continue to raise awareness of the human tragedy of unsafe abortion suffered by women, and particularly poor and marginalized women in low- and middle-income countries. The document highlights the urgency of preventing unsafe abortion and provides policy and programme recommendations for its prevention. The report also concluded that in countries where abortion laws were more liberal, rates of abortion were lower. Infertility due to unsafe abortion or maternal sepsis was identified as the fifth ranked disability, based on prevalence, in low- and middle-income countries for women aged 0–59 years. The Programme was also a key partner of the scientific team that developed preterm birth estimates – the team consisted of experts from HRP, the London School of Hygiene and Tropical Medicine and Save the Children. The input from HRP was particularly important in engaging and assisting countries in providing more data, understanding methods and supporting their capacity in improving measurement of preterm births. The Programme also contributed to the development of publications covering global estimates of stillbirths – National, regional, and worldwide estimates of stillbirth rates in 2009 with trends since 1995 (39) as well as estimates of the global burden of STIs in 2010 (40). The July 2010 progress report on the implementation of the WHO reproductive health strategy (41) noted in a survey of countries that: 85% reported integrating the WHO focused antenatal care approach in national reproductive health prorammes; 80% of the countries had registered magnesium sulfate for use in pre-eclampsia; and 50% had included emergency contraception as part of their family planning method mix. The percentage of countries reporting implementing new cervical screening protocols was somewhat lower (n total, 57 countries reported). The report also noted uneven progress in reduction of maternal mortality; that unmet family planning need was still high; that antenatal care and skilled birth attendance had increased in urban areas, with little change in rural areas; and that less than 50% of men aged 15–24 years reported using a condom when engaging in high-risk sexual behavior. HRP, in collaboration with UNFPA, defined a number of indicators on health outcomes and processes at national and subregional levels, as well as indicators to assess integration of HIV/AIDS and SRH programmes and services (42). From Table 8, it can be seen that over 70% of informants gave a rating of “1” or “2” for HRP’s work on monitoring global trends in SRH. Table 8 Effectiveness of HRP’s work in monitoring global trends Effectiveness Percentage of respondents 1 – highly effective 24.0 2 48.8 3 18.2 4 9.1 5 – not at all effective 0.0 Total (n = 121) 100.1a Conclusion HRP continues to play a major role in the monitoring and assessment of global trends in SRH, and this work is instrumental for evidence-based advocacy, the monitoring of progress HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 56 towards the achievement of the MDGs and other global targets and goals, and the initiation of national campaigns to address specific SRH issues. 9 The sexual and reproductive health/HIV research agenda What is HRP’s involvement in the SRH/HIV research agenda? There are clear bidirectional links between SRH and HIV-prevention policies and programmes. These can be used for example to: promote male and female condom use to prevent STIs, including HIV infection, and unintended pregnancy; decrease mother-to-child transmission (MTCT) of HIV; improve access to and uptake of key HIV and SRH services; improve access for people living with HIV to sexual and reproductive health care that protects their rights and respond to their needs; reduce HIV-related stigma and discrimination; increase the response to and prevention of GBV; improve coverage of underserved and marginalized populations; and decrease duplication of efforts, and thus competition for scarce resources. HRP has been actively pursuing collaborative work in this area over the last 5 years, including development of policy and programmatic guidance to address the SRH of people living with HIV; planning a multicountry study on the fertility intentions and unmet need for family planning among women living with HIV; and implementation research on integrated approaches for strengthening SRH and HIV services (see Chapter 6). 9.1 Selected highlights 9.1.1 Rapid assessment tool for sexual and reproductive health and HIV linkages: evidence review and recommendations HRP collaborated with IPPF, UNFPA, WHO, UNAIDS, the Global Network for People Living with HIV/AIDS, the International Community of Women with HIV/AIDS, and Young Positives, to develop a rapid assessment tool for SRH and HIV linkages at policy, system and service levels. The tool has promoted and enabled dialogue and collaboration between the HIV and SRH areas, which are often separated in national programme contexts. The tool has now been used in over 45 countries and the process, results and outcomes have been summarized in individual country briefs, which are then used as the basis for further action. The Kesho Bora Study The Kesho Bora Study (25) was a landmark piece of research led by HRP, using its own funds and leveraging additional funds from partners. It showed that giving mothers a combination of antiretroviral drugs during pregnancy, delivery and breastfeeding cut HIV infection in infants by 42%. The findings of the research increase the range of treatment options available to mothers living with the virus, and offers them hope that, if they so wish, they have a greater chance of breastfeeding their babies without passing on HIV. As a result of the study, Malawi and South Africa have adopted a new regime for preventing MTCT of HIV, and the findings have changed WHO’s recommendations on infant feeding and led to new drug-combination approaches in global efforts to eliminate MTCT of HIV. Collaboration with the Global Fund HRP continues to advocate for and provide assistance to programme countries in preparing proposals to the Global Fund to Fight AIDS, Tuberculosis and Malaria, particularly in the areas of: unmet needs and opportunities for linking SRH and HIV prevention and care; and opportunities to address violence against women and girls. The proportion of proposals with HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 57 an element of SRH has increased from under 40% in the first round to 90% and 65% respectively in the last two rounds. Collaboration with the WHO HIV/AIDS department Current and future work of the WHO HIV/AIDS department includes specific deliverables that require contributions from HRP. These include: a generic protocol for operational research to understand the performance and impact of use of dual point-of-care tests for HIV and syphilis; guidance on the evaluation, specifications and procurement of female condoms; tools for monitoring condom quality during storage; technical guidance on microbicides; clinical guides and tools for family planning for persons living with HIV; clinical guides and tools for counselling on testing for HIV in family planning clinics; policy and programme guidance for prevention of MTCT of HIV; guidance on infant feeding and transmission of HIV; guidelines on the prevention and management of STIs; an updated package for HIV treatment, prevention and care among young people; and evidence, tools and guidelines to address gender-based inequalities in HIV response for most-at-risk populations. Collaboration with UNAIDS Collaboration between HRP and UNAIDS includes areas such as: a discussion paper on male involvement in the prevention of mother-to-child transmission of HIV, which also indentified gaps in knowledge and areas for further research; the technical meeting on hormonal contraception and HIV risk (33), which provided clear guidance on this issue; development of a counselling tool on reproductive choices and family planning for people living with HIV (43); and guidance on issues such as male circumcision and HIV (44). As shown in Table 9, over 85% of respondents gave a rating of “1”, “2” or “3” for HRP’s effectiveness in contributing to the SRH/HIV research agenda. Comments from respondents suggested a need for HRP to be more proactive in this area (for example, as it had been over the issue of HIV infection and hormonal contraception); to forge stronger links with the WHO HIV/AIDS department and with UNAIDS (the latter perhaps now facilitated by UNAIDS becoming a permanent member of PCC); and for a greater focus on barriers and implementation issues relating to the integration of SRH and HIV programmes at country level. There was also some indication that the regional panels are increasingly generating more research on SRH/HIV issues. Table 9 Effectiveness of HRP in contributing towards the SRH/HIV research agenda Effectiveness Percentage of respondents 1 – highly effective 21.6 2 43.1 3 21.6 4 12.1 5 – not at all effective 1.7 Total (n = 116) 100.1a a This percentage over 100% is due to rounding. Conclusion The Programme continues to play an important role in shaping and implementing the SRH/HIV research agenda and should continue to strengthen links with HIV/AIDS research partners. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 58 10 Efficiency and effectiveness of HRP’s communication HRP communicates with its clients in a number of ways and through a number of mechanisms. Most of these involve its publications, which can be classified into nine separate categories as follows: • policy briefs, highlighting the policy implications of new research findings, such as Programmatic and research considerations for hormonal contraception for women at risk of HIV and women living with HIV – policy Implications (45); • monitoring and evaluation reports, such as Trends in maternal mortality: 1990 to 2010, WHO, UNICEF, UNFPA and The World Bank estimates (17); • advocacy documents, such as Providing the foundation for sexual and reproductive health. A record of achievement (46); • peer-reviewed articles, including systematic evidence reviews; • fact sheets, statements and information notes, such as Emergence of multi-drug resistant Neisseria gonorrhoeae – threat of a global rise in untreatable sexually transmitted infections (47); • newsletters, such as the Reproductive Health Update; • multimedia (videos, exhibits, posters, apps), such as the RHL DVD2, and HRP videos on YouTube; • clinical guidelines, such as: WHO recommendations for prevention and treatment of pre-eclampsia and eclampsia (15); • programme and policy documents, such as Beginning with the end in mind: planning pilot projects and other programmatic research for scaling up (48). For the main six categories, the Programme has produced 477 such documents over the past 5 years, of which further details are given in Table 10. By far the largest output of HRP is its peer-reviewed articles, and these increased from 52 in 2008 to 127 in 2011 and 80 in 2012. Respondents cited peer-reviewed publications and clinical guidelines as the most effective channels for communicating HRP’s results. Table 10 Numbers of communication documents produced by HRP, 2008 to 2012 Document type Number produced 2008 – 2012 Clinical guidelines 16 Programme and policy documents 51 Policy briefs 19 Monitoring and evaluation reports 19 Advocacy documents 15 Peer-reviewed articles 361 Total 481 2 The Reproductive Health Library is an electronic journal, cumulative over time, which publishes evidence syntheses on a variety of key SRH topics and includes comments on the application of the evidence to low-resource settings by experts from programme countries. It is also a conduit for dissemination of all the guidelines, norms and standards developed by RHR/HRP. The RHL CD- ROM was published in April 2011 with 193 Cochrane reviews. Since then, the online version has been expanded with two to three new and updated titles per month. RHL continued to rise in rank within the family of WHO web sites: of the 211 WHO unique web addresses, RHL moved from rank 15th in terms of number of sessions per month in 2010 to 7th in November 2011. At the end of 2012 it ranked 11th, but remained the second most visited site outside of WHO corporate pages. There are approximately 17 000 visits per month and over a million page views per month. The most downloaded document in 2011 was the SUPPORT project summary on the effectiveness of task-shifting programmes on maternal and child health outcomes, with 7800 downloads. The RHL Facebook social networking site was launched in late 2011 to increase exposure to a younger generation of SRH workers and is visited more than 50 000 times each month. Additionally, a YouTube channel was created in April 2012 and has received in excess of 65 000 video views per month since its launch. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 59 A somewhat subjective review of the titles of the peer-reviewed publications would appear to indicate that these are generated by both the Programme’s global research agenda and the support it provides to research-capacity strengthening. Some research papers are highly country specific, and others address issues that are clearly of more interest to the individual researchers than to the global agenda. The undertaking of research is an essential element of research-capacity strengthening, but such research cannot always be expected to reflect global research needs. Recommendation In future reporting, HRP should distinguish between peer-reviewed articles generated through its global agenda, and those generated from research-capacity-strengthening activities. This would provide more transparency and permit a greater understanding of the impact of the Programme’s work at both global and regional levels. HRP continues to undertake high-impact randomized trials, the type of research that answers key questions and provides the core evidence for clinical practice guidelines. This research is vital in determining the standards for the practice of evidence-based medicine and public health. A few examples of such high-impact work undertaken by the Programme since 2008 are given in Box 10. All of these led to new recommendations for clinical practice, which are being adopted by countries. Box 10. Headline examples of HRP’s high-impact research • Kesho Bora Study Group, de Vincenzi I. Triple antiretroviral compared with zidovudine and single-dose nevirapine prophylaxis during pregnancy and breastfeeding for prevention of mother-to-child transmission of HIV-1 (Kesho Bora study): a randomised controlled trial. Lancet Infectious Diseases, 2011, 11(3):171–180. Erratum in: Lancet Infectious Diseases, 2011, 11(3):159. • Meirik O et al; WHO Research Group on Postovulatory Methods of Fertility Regulation. Complications of first- trimester abortion by vacuum aspiration after cervical preparation with and without misoprostol: a multicentre randomised trial. The Lancet, 2012, 379(9828):1817–1824. • Althabe F et al; Guidelines Trial Group. A behavioral intervention to improve obstetrical care. New England Journal of Medicine, 2008, 358(18):1929–1940. • Gülmezoglu AM et al. Active management of the third stage of labour with and without controlled cord traction: a randomised, controlled, non-inferiority trial. The Lancet, 379(9827):1721–1727. • Warriner IK et al. Can midlevel health-care providers administer early medical abortion as safely and effectively as doctors? A randomised controlled equivalence trial in Nepal. The Lancet, 2011, 377(9772):1155–1561. • Widmer M et al. Misoprostol as an adjunct to standard uterotonics for treatment of post-partum haemorrhage: a multicentre, double-blind randomised trial. The Lancet, 2010, 375(9728):1808–1813. • Meirik O et al; IUD Research Group at the UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction. The frameless copper IUD (GyneFix) and the TCu380A IUD: results of an 8-year multicenter randomized comparative trial. Contraception, 2009, 80(2):133–141. • von Hertzen H et al; WHO Research Group on Post-ovulatory Methods of Fertility Regulation. Two mifepristone doses and two intervals of misoprostol administration for termination of early pregnancy: a randomised factorial controlled equivalence trial. British Journal of Obstetrics and Gynaecology, 2009, 116(3):381–389. • von Hertzen H et al; WHO Research Group on Post-ovulatory Methods of Fertility Regulation. Comparison of vaginal and sublingual misoprostol for second trimester abortion: randomized controlled equivalence trial. Human Reproduction, 2009, 24(1):106–112. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 60 10.1 Bibliometric analysis of HRP’s papers To provide more evidence of the quality and impact of the HRP’s work, the evaluation team requested the Programme to commission a bibliometric analysis of its peer-reviewed publications. The analysis was contracted out to Thomson Reuters in the UK and the report is briefly summarized in this section. Chapter 8 provides more information about the analysis. Citations to prior work are a normal part of publication, and reflect the value placed on the work by later researchers. Highly cited work is recognized as having a greater impact and being highly correlated with other qualitative evaluations of research performance. Citation analysis has to be used with some caution; it is more rigorous for science and technology and less so for social sciences and the humanities. Citations generally increase over time, so very recent publications will generally have fewer citations. Normalization is usually done by reference to relevant global averages for the particular field. The Thomson Reuters analysis was able to identify and match 1842 HRP publications in its database, of which the majority (89.6%) were peer-reviewed articles and reviews. The most frequent type of journal in which HRP research is published is journals focusing on contraception, obstetrics and gynaecology, and andrology. Obstetrics and gynaecology accounts for the highest share, at 40.9% of all papers published, and the citation impact for these articles is twice the world average. HRP articles published in journals dealing with general and internal medicine (such as The Lancet and The New England Journal of Medicine) and oncology had a citation impact of between two and three times the world average. Overall, the normalized citation impact of HRP publications has risen from an already impressive level of 1.42 during the period 1990 to 2007, to 2.14 between 2008 and 2011, indicating a significant improvement in the impact of the Programme’s research. The proportion of papers that are highly cited (those papers that belong to the world’s top 10% of the most cited papers relative to the journal category and year) also increased between these two periods, from 16.5% to 18.3%. Sixteen of the 20 most commonly used journals for HRP publications are in the top quartile, according to journal impact factor. Acceptance by such journals is an indicator that the publication is more likely to have a greater impact. This provides further evidence of the impact of the Programme. All these indicators of research performance are significantly above world averages and clearly reflect high-quality research, and research that is well regarded among the international research community. In terms of collaborations between countries, the study calculated collaboration indices at two different thresholds: at least one paper per year on average over the period, and at least one paper every two years on average over the period. For the higher-threshold index, there were no collaborations with low-income countries during the period 1990–2007, but for the period 2008–2011, five countries (Bangladesh, Burkina Faso, Nepal, Uganda, and Viet Nam) all published at least four collaborative papers with the Programme. For the lower-threshold index, there was an increased level of collaboration with some middle-income countries in Asia and Latin America over the two periods, but again the major change was the increased involvement of low-income countries, from one in the earlier period (Kenya) to eight in the period 2008–2011 (Bangladesh, Burkina Faso, Cambodia, Ethiopia, Kenya, Nepal, Uganda and Viet Nam). This analysis provides clear evidence of the increased involvement, particularly of low-income countries, in the Programme’s work. The Thomson Reuters analysis also carried out an author analysis. This indicated that over the period 1990–2011, the proportion of papers where all authors were from programme HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 61 countries fell from 23.2% to 13.7%, while at the same time, the proportion of papers where any author was from a programme country increased from 43.4% to 63.9%. A possible explanation for this is that as institutions matured over the years with the support provided by HRP for research-capacity strengthening, they have increasingly been part of the international collaborative research community, taking part in HRP’s global research agenda as well as the international research agendas of other organizations. Finally, since 2008, data have been collected on the countries of residence of first authors, and the analysis showed that during the period 2008–2011, 40% of all papers had a first author from a programme country. This is below the 2011 level for TDR, which currently stands at 61%, and reflects the need for the Programme to strengthen its efforts to involve institutions in programme countries. 10.2 HRP’s channels of communication The questionnaire asked for opinions about the effectiveness of various groups in communicating HRP’s products, using a scale of 1–5 (1: highly active, 5: minimally active). The majority of respondents gave negative ratings. The figures in parentheses indicate the proportion of respondents who gave a rating of “3” or lower in relation to the level of activity in disseminating the products of the Programme: WHO country offices (72.2%); WHO regional offices (66.3%); WHO headquarters (59.1%); regional advisory panels (52.6%); cosponsors (55.2%); and donors (58.2%). All these figures indicate considerable room for improvement on the part of cosponsors and donors in helping to communicate the publications of HRP. Many respondents felt that HRP needed to strengthen the advocacy, communication and dissemination of its products, in particular by developing a strategy that specifically targets end users in programme countries and involves the regional and country offices of WHO, as well as the other cosponsors, donors and civil society organizations (CSOs), amongst others. There was also an indication that the strategy should be linked to a stronger focus in the Programme on programmatic, implementation and social science research to better understand the introductory process and its barriers. Many respondents suggested that cosponsorship should become much more of a “two-way street”. As well as providing HRP with political and financial support, the cosponsors needed to become “champions” for the Programme’s products, playing a stronger role in disseminating the products and integrating them into their own development assistance programmes. Findings from the UNFPA survey, which also sought views on the effectiveness of different communication documents, were somewhat similar. As can be seen from Table 11, clinical guidelines, programme and policy documents, fact sheets and statements, advocacy documents, and policy briefs were the materials UNFPA offices felt were most effective in contributing to the improvement of national programmes. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 62 Table 11 Effectiveness of different types of communication documents Document type Percentage of UNFPA responses giving an effectiveness rating of “1”, “2” or “3” (1: highly effective; 5: not at all effective) Peer-reviewed articles 54.4 Advocacy documents 69.6 Clinical guidelines 75.8 Programme and policy documents 75.7 Training materials 63.6 Newsletters 45.4 Policy briefs 66.7 Global estimates 51.5 Fact sheets and statements 72.8 Conference presentations 54.5 RHL, CD-ROM 57.6 Press releases 57.6 In elaborating on their replies, there were many suggestions to improve communication between HRP and UNFPA country offices and national counterparts, the most common being the need to develop and keep up-to-date lists of key stakeholders at country level, such as national SRH managers, national NGOs, universities, training schools, CSOs and professional associations, and ensure that these groups were systematically kept informed of all new developments and products. Other suggestions included: greater use of social media channels; regional workshops for introductory activities; and web site linking. But the overall clear message was the need for HRP to strengthen its communication strategy to guide the dissemination of its products at country level, to better ensure their use. 10.3 Changes in RHR over the last 5 years and how these have affected HRP’s research communication and uptake HRP creates many valuable global SRH goods, but the ultimate value of these goods is in their utilization to improve the quality of SRH policies and programmes in countries, so that individuals can benefit. There is a continuum from research to action, of which research communication and uptake is an integral and essential component. Up until 2007, HRP had two direct mechanisms for promoting the use of its products in countries. These were: a well-funded Department for Programme Development in Reproductive Health (PDRH), and a well-funded Strategic Partnership Programme (SPP) with UNFPA, in the amount of over US$ 6 million between 2003 and 2007. An external evaluation of the SPP in 2007 (UNFPA unpublished document) found that it had used, and promoted with national authorities, a systematic process for the adaptation, adoption and introduction of evidence-based guidelines in national programmes. The SPP always involved governments, mainly ministries of health, from the outset (thus ensuring national ownership); translated evidence into practice; validated a model for use with other guidelines; and validated a model that could be drawn upon by other organizations for the introduction of their guidelines. With the end of the SPP in 2008, and the funding constraints in PDRH that followed from 2008 to 2011, brought about by changes in WHO core funding policies, the introduction of HRP’s products in countries waned, and the potential impact of its products was compromised. HRP started to lose a substantial amount of its ability to ensure that its products were translated into use at country level. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 63 Trends in income for PDRH are presented in Figure 3. PDRH income rose rapidly from US$ 9.2 million in 2000–2001 to US$ 25.0 million in 2006–2007, an increase of over 170%. During this period, its income consistently exceeded its full budget level. Since 2006–2007, PDRH income has fallen by around 16% to levels of US$ 20.4 million and US$ 21.0 million respectively in the last two biennia. In 2008–2009, income did not reach the contingency budget. In 2010–2011, income was between contingency and full budget levels. Figure 3 Trends in PDRH income Between 2006–2007 and 2010–2011, national governments decreased their contributions by US$ 6.6 million (38%), and UN system agencies, programmes and funds decreased their contributions by US$ 3.6 million (65%). Fortunately, foundations increased their contributions fourfold over the same period, to a level of US$ 8.2 million, to bring some stability to the PDRH funding situation. Major bilateral donors in 2006–2007, such as Denmark, Finland, the Netherlands, Norway, Sweden and the UK, combined to provide US$ 12.0 million of funding to PDRH. Subsequently, as requested by WHO, they included this funding in their core support to the WHO budget. However, they have not seen their previous PDRH contributions reflected in increased WHO Core Voluntary Contribution (CVC) support to PDRH. WHO CVC support to PDRH amounted to US$ 2.5 million in 2008–2009, and US$ 3.5 million in 2010–2011. This represents “lost” funding to PDRH of US$ 9.5 million and US$ 8.5 million respectively in the last two biennia. PDRH’s income declined by 16%; had previous funding mechanisms been maintained, it might have increased over the same period by more than 20%, to a level of around US$ 30 million. Fluctuations in income to PDRH from different donor groups present an additional challenge to the work and management of PDRH, since a large proportion of the new funding in the last 5 years is specified rather than flexible. As can be seen from Table 12, flexible funding to PDRH has fallen dramatically in the last three biennia, from 56.0% of total income to 26.3% of total income. Specified funds now make up almost three quarters of income to PDRH. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 64 Table 12 PDRH flexible and specified income, 2006–2011 Biennium Flexible income Specified income Total US$ million % US$ million % US$ million % 2006–2007 14.0 56.0 11.0 44.0 25.0 100.0 2008–2009 6.5 31.7 14.0 68.3 20.5 100.0 2010–2011 5.5 26.3 15.4 73.7 20.9 100.0 One of the major consequences of this trend is that, over the past 6 years, flexible income has increasingly been used to pay for staff salaries. In 2008, PDRH had 35 posts. In 2012, it was reduced to 22 posts. Since 2010, all flexible PDRH funding has been used to pay staff salaries. With no flexible funds for activities, these staff have continued to implement projects funded by individual donors, rather than doing the core work of RHR – promoting and introducing HRP’s products in countries. A further consequence of the loss of core PDRH funding is that, while up until 2008, the costs of STAG and GAP were shared between HRP and PDRH, since 2009 this has not been possible, and the full cost has been underwritten by HRP. As can be seen in Table 13, the vast majority of respondents gave a very strong indication of the negative impact of the reduction in PDRH funding on the promotion and utilization of HRP’s products. More than 70% gave a rating of “4” or “5”. Table 13 Respondents’ views on how the reduced funding to PDRH has affected the promotion and utilization of HRP’s products Rating Percentage of respondents 1 – not at all 3.6 2 8.3 3 17.9 4 36.9 5 – very negatively 33.3 Total (n = 84) 100.0 Conclusion HRP continues to carry out the research and systematic reviews that enable the production of evidence-based policies, programme interventions and clinical guidelines for SRH. Objective indicators of HRP’s research performance are significantly above world averages and clearly reflect high-quality research, and research that is well regarded among the international research community. However, the Programme needs to do more to increase the direct and lead involvement of programme countries in its work. The Programme also needs to be a global leader in efforts to develop and evaluate more effective ways of communicating and introducing the knowledge it develops into SRH policies and programmes. However, with the end of the SPP and the severe cutback in funding to PDRH, the Programme has lost a substantial portion of its ability to ensure the communication and uptake of its products. Over the past few years, some ad hoc solutions have been used to address this issue, but none of them were universally acceptable or viable in the longer term. It is thus a matter of urgency that donors and cosponsors find a more permanent solution. The responsibilities of HRP are: first to do the research, review the evidence and prepare the products; second, to have an effective communication strategy that ensures that all its products are communicated to all those who need to be aware of them; and third, to have an uptake strategy that enables the Programme to demonstrate how to introduce some of its more critical products into a limited number of programme countries, and to assess their HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 65 potential or actual impact on SRH. In order to be accountable for this work, HRP needs sufficient funds in its budget to implement all these activities. This could be achieved in a number of ways. Donors could return to earmarking contributions to PDRH, thus restoring its “lost” funding, or they could request the Programme to use, say, 20% of the funds for research on “research communication and uptake”. HRP has a long- established practice of spending one dollar on strengthening research capacity for every two dollars spent on research; it may now need to review these proportions and include a percentage of the budget for “research communication and uptake”. The larger task of introduction and uptake falls to national governments, the cosponsors, donors, foundations, CSOs and other. All these groups have internal responsibilities to use the uptake “success stories” demonstrated by HRP, and to channel their funds for the same purpose in the countries where they work, to ensure wider and greater impact of the Programme’s products, as well as greater value of their own investments. Such a process would ensure that the value of HRP’s products is maximized. It would also enable the Programme to more clearly demonstrate the relationship between the funds invested in an area of the work and the results of that investment at country level. It would also be possible eventually to track the cascading effect of the Programme’s work as its products are picked up and used by cosponsors, bilateral donors and others, in their programmes of development assistance. Recommendations There is a need for HRP to develop and invest in a new communication strategy, which explores innovative ways of packaging and disseminating HRP’s research findings and other products for use in strengthening national SRH policies and programmes. The strategy should consider the role of knowledge intermediaries and gatekeepers of change, and that different products will require very different approaches. Subsequent communication workplans should identify clear deliverables and associated indicators. HRP needs to develop, invest in, and implement a strategy for the utilization of its key products into a limited number of countries, to demonstrate their potential or actual impact, and to thereby leverage and guide the use of the funds of national governments, cosponsors, bilateral agencies, CSOs, foundations and others in their support to national SRH programmes. The PCC will need to provide guidance on the source of funding for HRP’s communication and utilization work. HRP donors and cosponsors need to review and strengthen their systems and processes for utilizing HRP’s products in their own programmes of development assistance, in order to maximize the value of HRP’s global goods. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 66 References 1. World Health Organization, Department of Reproductive Health and Research. Reproductive health strategy: to accelerate progress towards the attainment of international development goals and targets. Geneva, World Health Organization, 2004 (WHO/RHR/04.8) (http://whqlibdoc.who.int/hq/2004/WHO_RHR_04.8.pdf, accessed 19 December 2012). 2. Global strategy for the prevention and control of sexually transmitted infections: 2006– 2015. Breaking the chain of transmission. Geneva, World Health Organization, 2007. 3. United Nations Secretary-General. Global strategy for women’s and children’s health. New York, United Nations, 2010 (http://www.who.int/pmnch/topics/maternal/201009_globalstrategy_wch/en/index.h tml, accessed 19 December 2012). 4. Remme JHF et al. Defining research to improve health systems. Plos Medicine, 2010, 7(11):e1001000. 5. Department of Reproductive Health Research, World Health Organization. Medical eligibility criteria for contraceptive use, 4th ed. Geneva, World Health Organization, 2009. 6. World Health Organization, Department of Reproductive Health and Research. Selected practice recommendations for contraceptive use, 2nd ed. Geneva, World Health Organization, 2004. 7. Johns Hopkins Bloomberg School of Public Health/Center for Communication Programs and World Health Organization. Family planning: a global handbook for providers. Geneva, World Health Organization, 2011. 8. World Health Organization, Department of Reproductive Health and Research. Decision-making tool for family planning clients and providers. A resource for high- quality counselling. Geneva, World Health Organization, 2005. 9. World Health Organization, Department of Reproductive Health and Research. Medical eligibility criteria wheel for contraceptive use. 2008 update. Geneva, World Health Organization, 2009. 10. World Health Organization, Department of Reproductive Health and Research. WHO laboratory manual for the examination and processing of human semen, 5th ed. Geneva, World Health Organization, 2010. 11. WHO model list of essential medicines, 17th ed. Geneva, World Health Organization, 2011. 12. World Health Organization, Department of Reproductive Health and Research. Evaluating the quality of care for severe pregnancy complications: the WHO near-miss approach for maternal health. Geneva, World Health Organization, 2011. 13. WHO multicountry survey of maternal and newborn health 2010–2012. Geneva, World Health Organization. 14. World Health Organization, Department of Reproductive Health and Research. WHO recommendations for induction of labour. Geneva, World Health Organization, 2011. 15. WHO recommendations for prevention and treatment of pre-eclampsia and eclampsia. Geneva, World Health Organization, 2011. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 67 16. World Health Organization, Department of Reproductive Health and Research. Managing complications in pregnancy and childbirth. A guide for midwives and doctors. Geneva, World Health Organization, 2007. 17. WHO, UNICEF, UNFPA and the World Bank. Trends in maternal mortality: 1990 to 2010. WHO, UNICEF, UNFPA and the World Bank estimates. Geneva, World Health Organization, 2012. 18. World Health Organization, Department of Reproductive Health and Research. Safe abortion: technical and policy guidance for health systems, 2nd ed. Geneva, World Health Organization, 2012. 19. World Health Organization, Department of Reproductive Health and Research. Sexually transmitted and other reproductive tract infections: a guide to essential practice. Geneva, World Health Organization, 2005. 20. World Health Organization, Department of Reproductive Health and Research. Strategies and laboratory methods for strengthening surveillance of sexually transmitted infections. Geneva, World Health Organization, 2012. 21. World Health Organization, Department of Reproductive Health and Research. Use of cryotherapy for cervical intraepithelial neoplasia. Geneva, World Health Organization, 2011. 22. World Health Organization, Department of Reproductive Health and Research. Preventing gender-biased sex selection. An interagency statement, OHCHR, UNFPA, UNICEF, UN Women and WHO. Geneva, World Health Organization, 2011. 23. UNAIDS, UNDP, UNFPA, UNHCR, UNICEF, UNIFEM, WHO, FIGO, IOM, MWIA, WCPT, WMA. Global strategy to stop health care providers from performing female genital mutilation. Geneva, World Health Organization, 2010 (WHO/RHR/10.9). 24. World Health Organization, Department of Reproductive Health and Research. Transforming health systems: gender and rights in reproductive health. A training manual for health managers. Geneva, World Health Organization, 2001 (WHO/RHR/01.29) 25. Kesho Bora Study Group, de Vincenzi I. Triple antiretroviral compared with zidovudine and single-dose nevirapine prophylaxis during pregnancy and breastfeeding for prevention of mother-to-child transmission of HIV-1 (Kesho Bora study): a randomised controlled trial. Lancet Infectious Diseases, 2011, 11(3):171–180. Erratum in: Lancet Infectious Diseases, 2011, 11(3):159. 26. UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP). Mid-level health-care providers are a safe alternative to doctors for first-trimester abortions in developing countries. Geneva, World Health Organization, 2008 (WHO/RHR/HRP/08.15). 27. Federal Ministry of Health. Saving newborn lives in Nigeria: newborn health in the context of the Integrated Maternal, Newborn and Child Health strategy, 2nd ed. Abuja, Federal Ministry of Health, Save the Children, Jhpiego, 2011. 28. Optimizing health worker roles to improve access to key maternal and newborn health interventions through task shifting. Geneva, World Health Organization, 2012. 29. OHCHR, UNAIDS, UNDP, UNECA, UNESCO, UNFPA, UNHCR, UNICEF, UNIFEM, WHO. Eliminating female genital mutilation. An interagency statement. Geneva, World Health Organization, 2008. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 68 30. Global strategy to stop health-care providers from performing FGM. Geneva, World Health Organization, 2010 (WHO/RHR/10.9). 31. World Health Organization and London School of Tropical Medicine. Preventing intimate partner and sexual violence against women: taking action and generating evidence. Geneva, World Health Organization, 2010. 32. World Health Organization Department of Reproductive Health and Research. Preventing gender-biased sex selection: an interagency statement. Geneva, World Health Organization, 2011. 33. World Health Organization, Department of Reproductive Health and Research. Hormonal contraception and HIV. Technical statement. Geneva, World Health Organization, 2012 (WHO/RHR/12.08 ). 34. The WHO application of ICD-10 to deaths during pregnancy, childbirth and puerperium: ICD MM. Geneva, World Health Organization, 2012. 35. WHO guidelines for the management of postpartum haemorrhage and retained placenta. Geneva, World Health Organization, 2009. 36. Mikolajczyk RT et al. A global reference for fetal-weight and birthweight percentiles. The Lancet, 2011, 377:1855–1861. 37. International statistical classification of diseases and related health problems, 10th revision. Geneva, World Health Organization, 2010 (http://apps.who.int/classifications/icd10/browse/2010/en, accessed 1 January 2013). 38. World Health Organization, Department of Reproductive Health and Research. Unsafe abortion. Global and regional estimates of the incidence of unsafe abortion and associated mortality in 2008, 6th ed. Geneva, World Health Organization, 2011. 39. World Health Organization, Department of Reproductive Health and Research. National, regional, and worldwide estimates of stillbirth rates in 2009 with trends since 1995. Geneva, World Health Organization, 2011 (WHO/RHR/11.03). 40. World Health Organization, Department of Reproductive Health and Research. Prevalence and incidence of selected sexually transmitted infections. Chlamydia trachomatis, Neisseria gonorrhoeae, syphilis and Trichomonas vaginalis. Methods and results used by WHO to generate 2005 estimates. Geneva, World health Organization, 2011. 41. World Health Organization, Department of Reproductive Health and Research. Reproductive health strategy: to accelerate progress towards the attainment of international development goals and targets. Geneva, World Health Organization, 2004 (WHO/RHR/04.8) (http://whqlibdoc.who.int/hq/2004/WHO_RHR_04.8.pdf, accessed 19 December 2012). 42. World Health Organization, Department of Reproductive Health and Research, UNICEF. National-level monitoring of the achievement of universal access to reproductive health: conceptual and practical. Considerations and related indicators. Geneva, World Health Organization, 2008. 43. Reproductive choices and family planning for people living with HIV: counselling tool. Geneva, World Health Organization, 2006. 44. Consultation to review manufacturing, clinical and regulatory requirements for male circumcision devices to support programme expansion in high HIV incidence settings in Africa. Meeting report, Nairobi, Kenya 11–12 March 2009. Geneva, World Health HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 69 Organization, 2009 (http://www.who.int/hiv/pub/malecircumcision/ meetingreport_mar09/en/, accessed 29 January 2013). 45. World Health Organization, Department of Reproductive Health and Research. Programmatic and research considerations for hormonal contraception for women at risk of HIV and women living with HIV. Policy Implications. Geneva, World Health Organization, 2012 (WHO/RHR/12.09). 46. UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP). Providing the foundation for sexual and reproductive health. A record of achievement. Geneva, World Health Organization, 2008 (WHO/RHR/HRP/08.13). 47. World Health Organization, Department of Reproductive Health and Research. Emergence of multi-drug resistant Neisseria gonorrhoeae – threat of a global rise in untreatable sexually transmitted infections. Geneva, World Health Organization, 2011 (WHO/RHR/11.14). 48. World Health Organization, Department of Reproductive Health and Research – ExpandNet Beginning with the end in mind: planning pilot projects and other programmatic research for scaling up. Geneva, World Health Organization, 2011. HRP External Evaluation 2008−2012 (DRAFT) Chapter 2 Page | 70 Page | 71 Chapter 3 Efficiency and effectiveness of HRP’s governance, management and administration Nicholas Dodd Consultant, Sexual and Reproductive Health Page | 72 Page | 73 Contents 1 Introduction . 75 2 Funding and fundraising . 75 2.1 Overall funding . 75 2.2 Designated funding . 76 2.3 Leveraged funding . 78 2.4 Income from royalties . 78 2.5 HRP fundraising initiatives and achievements, 2008 to 2012 . 78 3 Financial management . 80 4 Cosponsorship . 80 5 Governance. 82 5.1 The Policy and Coordination Committee . 82 5.2 The Scientific and Technical Advisory Group . 84 5.3 The Gender and Rights Advisory Panel. 86 5.4 The Research Project Review Panel . 87 5.5 Costs of HRP governance . 89 6 Management and administration . 91 6.1 Managing for results . 91 6.2 Managing research grants . 93 6.3 Managing research . 94 Reference . 96 Annex 1: Trends in HRP income from royalties, 2000–2011 . 97 Annex 2: Summary of HRP resource-mobilization activities 2008–2012 . 98 Annex 3: HRP governance structure . 100 Page | 74 Abbreviations ERC Ethics Review Committee FEW-OHRP Federal Wide Assurance for the Protection of Human Subjects – Office for Human Research Protections Database GAP Gender and Rights Advisory Panel HRP UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction ICPD International Conference on Population and Development JCB Joint Coordinating Board OWER Organization-Wide Expected Result (WHO) PCC Policy and Coordination Committee PDRH Programme Development in Reproductive Health RAP regional advisory panel RHR WHO Department of Reproductive Health and Research RP2 Research Project Review Panel SERG Scientific and Ethics Review Group SPP UNFPA–WHO Strategic Partnership Programme STAG Scientific and Technical Advisory Group TDR Special Programme for Research and Training in Tropical Diseases UK United Kingdom of Great Britain and Northern Ireland UN United Nations UNAIDS Joint United Nations Programme on HIV/AIDS UNDP United Nations Development Programme UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USA United States of America WHO World Health Organization HRP External Evaluation 2008−2012 Chapter 3 Page | 75 1 Introduction This chapter does not attempt to review and evaluate all aspects of the governance, management and administration of HRP (the United Nations Development Programme (UNDP)/United Nations Population Fund (UNFPA)/United Nations Children’s Fund (UNICEF)/World Health Organization (WHO)/World Bank Special Programme of Research, Development and Research Training in Human Reproduction). Given the time and resources available, only a number of the more central issues have been examined. The chapter examines the following: funding and fundraising; financial management; cosponsorship; the functioning of the main governance bodies of HRP: the Policy and Coordination Committee (PCC), the Scientific and Technical Advisory Group (STAG), the Gender and Rights Advisory Panel (GAP), and the Research Project Review Panel (RP2); and selected aspects of programme management, including managing for results, managing research grants and managing research. 2 Funding and fundraising HRP’s level of income both reflects and affects its ability to achieve its goals and objectives. Fundraising is one of the prime responsibilities of the director of the Programme, but a responsibility that is subject to many external factors over which he or she has little control. Over the last 5 years, the most important of these has probably been the global financial climate, which has inevitably affected the development budgets of both governments and United Nations (UN) funds, programmes and agencies. 2.1 Overall funding shows trends in HRP’s income between 1996 and 2011, and the relation between its income and its approved budgets at both full and contingency levels. Income to the Programme declined each biennium between 1996 and 2005, and then increased by 80% in the next biennium, 2006–2007, to over US$ 45 million. In the last two biennia, income declined by around 10%, but remained relatively stable at a little in excess of US$ 40 million. Current income projections for the 2012–2013 biennium predict an increase to over US$ 45 million. Figure 1 HRP biennial income, and full and contingency budget levels, 1996–2011 (US$ millions) HRP External Evaluation 2008−2012 Chapter 3 Page | 76 Between 1996 and 2005, income consistently fell below the approved budget at both full and contingency level, and only in the 2006–2007 biennium did income exceed the budget. In the last two biennia, income has remained in between the full and contingency budget levels, though in reality closer to the contingency budget. In the current biennium, 2012–2013, the income and full budget gap is projected to continue at a shortfall of around 10%. Figure 2 shows trends in HRP income by major donor groups – governments; cosponsors; foundations; and civil society/private sector/other. Governments continued to provide the major portion of HRP’s income. Over the period 1996 to 2011, governments provided an average of 58% of HRP’s income. This has increased more recently, and in the 2010–2011 biennium it reached 72% of total income. Funding from cosponsors has declined over the last 15 years by around 50%, from US$ 14 million in 1996–1997 to around US$ 7 million in 2010– 2011. Contributions from foundations, though somewhat variable on a year-to-year basis, have become a significant source of HRP’s income. Figure 2 HRP biennial income by major donor grouping, 1996 to 2011 (US$ millions) 2.2 Designated funding The Programme prepared and the PCC approved guidelines for the acceptance of designated funding in 1998, and trends in core and designated funding from 2000 to the present are shown in Figure 3. Over the years, designated funding has continue to increase, and continues to be an important source of HRP’s income, currently running at around 20% of total income. In the 2000–2001 biennium, HRP had an income of US$ 33.3 million, of which US$ 858 248 (2.6%) was designated funding. After deducting costs for staff salaries1, general technical 1 In the 2006–2007 biennium, the Programme had 53 posts. In the 2010–2011 biennium, this had increased by only one to 54 posts. The ratios between professional and support staff have also changed positively, from being almost 1:1 up until 2005 to a ratio of around 2:1 in the 2010–2011 biennium. The Programme kept its staffing level more or less in line with its income over the period of the evaluation. However, the 53 posts in the 2006–2007 biennium cost the Programme US$ 13.4 million, whereas the 54 posts in the 2010–2011 biennium cost the Programme US$ 25.2 million, an increase of almost 90%. Approximately one third of this increase was due to changes in the exchange rate between the US dollar and the Swiss Franc, and two thirds was due to increases in salaries, post adjustment, and an 8.5% increase in WHO overhead payments with the introduction of the “post occupancy” charge. Currently, the Programme has some 20 vacancies. Seventy-five per cent of respondents gave the Programme a rating of “3” or better on its effectiveness in managing its human resources (scale 1 to 5 where 1 is highly effective and 5 is not at all effective). Many noted that the Programme is not completely independent in this regard, and is bound to the policies and procedures of WHO. HRP External Evaluation 2008−2012 Chapter 3 Page | 77 activities and programme management (US$ 13.0 million), the Programme had around US$ 20 million for research activities, of which more than 95% was undesignated. In the 2010–2011 biennium, HRP had an income of US$ 41.7 million, of which US$ 7.6 million (18.2 %) was designated. After again deducting costs for salaries, general technical activities and programme management (US$ 28.0 million), the Programme had around US $ 13.7 million for research activities, of which US$ 5.1 million (37.2%) was designated. Figure 3 HRP designated and core funding 2000–2011 (US$ millions) Thus, the total funds available for HRP research activities have decreased by 31.5% over the last 10 years; undesignated funding for research activities has decreased by 58 per cent, and designated funding for research activities has increased by a factor of almost six. In 2000–2001, every staff dollar implemented US$ 1.5 of activities; in 2010–2011, every staff dollar implemented US $ .50 cents of activities. Conclusion Core funding and core funds available for research have both declined significantly. The importance of core funding should not be underestimated, since it enables the HRP research agenda to be driven by needs and priorities that are independently and consensually identified; by the products that countries need to improve the quality of their sexual and reproductive health policies and programmes; and by scientific and technical assessments of their feasibility and likely impact. The unavoidable increases in the costs of staffing compounded by the weakening of the US dollar against the Swiss Franc have reduced the efficiency of the Programme over the last ten years. The increased proportion of designated income has also caused the Programme additional work in terms of developing proposals, and then monitoring and reporting on them. To regain its efficiency, the Programme needs an increase in undesignated funding, and needs to contract out a greater proportion of its work (see also section 6.3). Recommendation All donors to HRP should reflect on the importance of providing the Programme with undesignated funding, and, wherever possible, provide undesignated funds on a multiyear basis. Where this is not possible, the current practice of providing designated funds for specific HRP External Evaluation 2008−2012 Chapter 3 Page | 78 items of HRP’s already approved workplan and budget should continue. The Programme should explore the possibility of additional funding from new foundations located outside the United States of America (USA). 2.3 Leveraged funding Table 1 shows trends in leveraged funding for the last six biennia. Leveraged funding includes those projects in which HRP develops a partnership involving cost sharing or contributions in kind, or those projects that are brokered in their entirety by another agency. Data on HRP research projects and other activities that have involved this type of partnership are collected each year and the total value of the HRP investment in these projects since 2000 has been US $13.8 million. It is estimated that the Programme’s partners contributed an additional US$ 51.9 million to the same projects, thus representing a “leveraging” of almost US $4 for every dollar invested by the Programme. Table 1 Trends in HRP leveraged funding, 2000 to 2011, US$ millions Biennium HRP Funding Leveraged funding Ratio 2000/2001 2 317 410 9 676 068 4.2 2002/2003 3 723 679 12 926 904 3.5 2004/2005 847 350 3 492 713 4.1 2006/2007 1 355 100 6 144 003 4.5 2008/2009 1 908 685 7 095 800 3.7 2010/2011 3 691 670 12 535 804 3.4 Total 13 843 894 51 871 292 3.7 While direct support to HRP is critical, the catalytic role that the Programme can play by providing support to initiatives in sexual and reproductive health that are funded or implemented by partners is also important, and demonstrates an added value for every dollar invested in HRP. 2.4 Income from royalties HRP income from royalties continues at a little under 3% of total income, as can be seen from the table in Annex 1. 2.5 HRP fundraising initiatives and achievements, 2008 to 2012 HRP developed its first resource-mobilization strategy in 2006, made possible through a capacity-strengthening grant from The David and Lucile Packard Foundation. A professional fundraising consultant was hired to advise on the strategy, which included the development of a standardized communication and follow-up approach with donors; donor roundtables and regular briefings; developing marketing materials and key messages; and strengthened resource-mobilization capacity of all HRP staff. The process culminated in an all-day retreat for professional staff of the WHO Department of Reproductive Health and Research (RHR), to discuss application of the strategy in their day-to-day work. This initiative, which also resulted in a full-time staff position for fundraising, laid the groundwork for HRP resource mobilization up until the present. 2008–2012 was a period of both global fiscal crises and WHO financial reform, both of which brought threats of reduced income to the Programme. Despite the fact that the number of donors to the Programme declined between 2009 and 2011, HRP was able to maintain its overall level of support. For 2012, three new donors are expected to raise the total number of donors to the Programme from 22 to 25. HRP External Evaluation 2008−2012 Chapter 3 Page | 79 The Programme is also currently hoping to consolidate a number of resource-mobilization initiatives. For example, the governments of Norway and Flanders have committed a substantial increase in 2012; in 2012 the US Government is providing a core contribution for the first time since 2005; the governments of Switzerland and the United Kingdom of Great Britain and Northern Ireland (UK) have both indicated a willingness to discuss increased as well as multiyear contributions in the future; and other key donors, including the Bill and Melinda Gates Foundation, David and Lucile Packard Foundation, an anonymous donor, and the Swedish Government, have committed to maintain or possibly increase their support. The current financial position for the Programme seems very positive, and income for the 2012– 2013 biennium is predicted to reach a level in excess of all previous biennia. In addition, in 2011 PCC proposed the establishment of a resource-mobilization subcommittee. This was endorsed by PCC in 2012, and is expected to further support the Programme’s resource-mobilization work, and perhaps initiate other activities. A more detailed summary of all HRP’s resource-mobilization activities can be found in Annex 2. Respondents’ views on funding and fundraising are shown in Table 2. Over 40% of respondents gave a rating of “1” or “2” for HRP’s work on strengthening its fundraising since 2008, and over 70% gave a rating of “3” or better (scale 1 to 5 where 1 is highly effective and 5 is not at all effective). In commenting on their ratings, a large number of respondents indicated that the single most important way for the Programme to secure additional funding was to document and communicate the utilization, and thus potential impact, of its products in programme countries, and particularly how these help achieve global goals in sexual and reproductive health. In addition, respondents advised that HRP should clearly identify those products that would have a significant potential impact in programme countries, but that could not be developed owing to insufficient funding. This would highlight the advances and opportunities that would be lost whenever income fell below full budget level. Table 2 Respondents’ views of HRP’s work on strengthening fundraising since 2008 Rating Percentage of respondents 1 – highly effective 10.9 2 30.4 3 30.4 4 26.1 5 – not at all effective 2.2 Total (n = 46) 100.0 A number of other respondents felt that the Programme needed to continue its outreach and engagement with foundations and global health initiatives to seek additional funds. As previously mentioned in Section 10 of Chapter 2, the vast majority of key respondents felt that the reduction in funding of the Programme Development in Reproductive Health (PDRH) has had a very negative effect on the promotion and use of HRP’s products, and that this has only been exacerbated by the termination of the UNFPA–WHO Strategic Partnership Programme (SPP) in 2008. Both these factors have somewhat reduced the value of the Programme’s products, and this is an issue that must be addressed as a matter of urgency. Conclusion The Programme should be commended on its resource-mobilization efforts. During a period of global financial crisis, when many organizations have experienced severe funding cuts, the progamme’s resource-mobilization initiatives have enabled it to maintain income at over US$ 40 million for the last two biennia. The projected income for the current (2012–2013) HRP External Evaluation 2008−2012 Chapter 3 Page | 80 biennium is expected to surpass all previous biennia. While designated funding has become a significant proportion of total income, core funding is crucial for the Programme to continue to be driven by independently identified needs and priorities. Recommendation HRP needs to continue to build on the success of its resource-mobilization work and strengthen it further by demonstrating and communicating the utilization of its products in programme countries, their potential impact, and how this helps the achievement of global targets in sexual and reproductive health. 3 Financial management As demonstrated in Section 2, there have been major fluctuations in HRP’s income over the past 5 years. These fluctuations are compounded by the timing of the receipt of funds, with much income being received towards the end of each year. A number of respondents felt the Programme should follow up more frequently with donors, in order to expedite their contributions as early as possible during the year. PCC generally views HRP’s management of funds very positively, and more than 67% of respondents gave a rating of “1” or “2” for the efficiency of the Programme’s management of its financial resources, and a further 25% gave a rating of “3”. Some respondents felt that more communication on financial matters should take place between PCC meetings (though this is already being done with the standing committee), and that HRP needed to continue to be very transparent in distinguishing between financial reporting of HRP and PDRH. Conclusion The Programme has continued to manage its financial resources in an efficient and effective manner. 4 Cosponsorship HRP cosponsorship is defined in the Memorandum of Understanding (see Chapter 1 Annex 2) as an expression of commitment rather than a legally binding relationship. The original cosponsors were UNDP, UNFPA, WHO and the World Bank, and these have remained so since HRP became a cosponsored programme in 1988. However, two important developments have taken place since 2008. First, UNDP increased its involvement as a cosponsor, playing a more active role in both the standing committee and PCC, and returning as a financial contributor with a grant of US$ 120 000 covering the years 2009 to 2011. UNDP continues to be an important global promoter of both the links between sexual and reproductive health, HIV/AIDS and development, and the importance of the sexual and reproductive health targets to the overall achievement of the Millennium Development Goals (MDGs). As the lead UN coordinating agency at country level, UNDP has an increasing responsibility to provide the platform for strengthening coordination of the UN system in providing assistance to national sexual and reproductive health programmes, utilizing the evidence base provided by HRP. Secondly, the Programme has successfully attracted a new cosponsor, the United Nations Children’s Fund (UNICEF). This was agreed at the PCC meeting in June 2012, and hopefully will result in future financial contributions. It also strengthens the sphere of partnership and influence of the Programme, and further expands opportunities for the promotion and use of HRP’s products in programme countries, especially in the context of the one UN. UNICEF officially became a cosponsor in December 2012. HRP External Evaluation 2008−2012 Chapter 3 Page | 81 On a related topic, the Joint United Nations Programme on HIV/AIDS (UNAIDS), unable to become a cosponsor because it is itself a cosponsored programme, nevertheless was accepted by PCC in 2012 as a new permanent member of PCC, joining the International Planned Parenthood Federation in that capacity. This development strengthens HRP’s potential synergies in both basic and programme research where sexual and reproductive health and HIV overlap. Also, again, it strengthens partnerships at the country level for the application of research findings and technical guidance. The cosponsors of HRP make up the standing committee, and this normally meets three times a year. A typical schedule would be once at the time of the PCC meeting, once by video- or teleconferencing, and once at the headquarters of one of the cosponsors. This enables the costs of the standing committee to be very well controlled, cosponsors pay for themselves, and the cost to the HRP secretariat is a mere 1% of total governance costs. However, more importantly, the meetings at cosponsor agency headquarters give the opportunity for the HRP director to interact with executive directors and other senior staff of the cosponsors, as well as to provide information on the work of the Programme and its outputs, at open cosponsor staff meetings. This will be a continuing opportunity for both updating on new developments, and reinforcement of the existence and importance of HRP’s work, particularly in view of the normal cycles of cosponsor staff rotations. These meetings further represent an opportunity to discuss stronger cosponsor engagement in advising on future research priorities; stronger support from cosponsors for use of the Programme’s products in countries; and future financial contributions from cosponsors, which have waned considerably in recent years. As can be seen from Table 3, the World Bank has been the most consistent and strongest financial support of HRP over the last decade, providing US $22 million or 49.0% of the total support from the four cosponsors. Next is UNFPA, providing US$ 14.2 or 31.6% of cosponsor support. Between them, these two cosponsors provide over 80% of the financial support from cosponsors, while WHO provides almost all of the remaining 20 per cent. Table 3 Trends in funding by HRP cosponsors, 2000–2011 Biennium Cosponsor contribution (US$ millions) UNDP UNFPA WHO World Bank 2000/2001 0 3.0 1.3 4.0 2002/2003 0 2.6 1.1 3.0 2004/2005 0 3.7 2.1 4.0 2006/2007 0 2.2 1.6 4.0 2008/2009 0.06 1.0 1.5 3.5 2010/2011 0.06 1.8 1.3 3.5 Total 2000–2011 0.12 14.2a 8.6 22.0a a All figures have been rounded up. Overall totals in the final row are correct, though they do not correspond exactly with totals derived from the figures in the body of the table because of rounding errors. While a number of respondents to the questionnaire felt that the political and financial support provided by cosponsors was sufficient, a larger number felt that it could be much improved. This group felt that financial support to the Programme from cosponsors should be increased, that cosponsors should also become stronger champions for HRP, and that new partnerships such as the previous SPP with UNFPA should be developed to add more value to the Programme’s products. Respondents felt that the joint UN statements initiated by the Programme, but issued under the names of the cosponsors, and sometimes additional UN funds agencies and programmes, was a clear example of the added value of cosponsorship. HRP External Evaluation 2008−2012 Chapter 3 Page | 82 On the possibility of additional cosponsors, responses were somewhat mixed. A number of respondents cautioned against further increasing the number of cosponsors because it would cost the Programme more time investment in exchange for perhaps little additional income and possibly greater influence on the research agenda. However, the majority was in favour of increasing the number of cosponsors, providing they were willing to commit funds and the necessary time to help guide the Programme. Most respondents in this group indicated that UN Women and the UN High Commission for Refugees could be the next two most logical organizations for the Programme to approach. Finally, respondents felt that the Programme needed to further strengthen and continue its direct communication with the executive directors and senior management of the cosponsors, in order to promote all aspects of cosponsorship: funding, political support and partnerships. Conclusion Cosponsorship continues to be considered a very important element of the fundamental structure of HRP. However, it needs to be reinvigorated, as much by the cosponsors themselves as by the Programme. Recommendation HRP and the cosponsors need to strengthen their engagement, developing clear plans and mechanisms to use the programmatic experience and networks of the cosponsors to help identify key research questions and needs for policy, programmatic and technical guidance, and to use their programmes and networks to promote and expand the use of HRP’s products in countries. A progress report should be presented to PCC after 2 years. The Programme should, somewhat cautiously, explore additional cosponsors. 5 Governance This section of the evaluation examines the functioning of PCC, STAG, GAP and RP2. A review of the regional advisory panels (RAPs) can be found in the case study on research-capacity strengthening in Chapter 5. A later section of this chapter reviews the costs of HRP governance, and a schematic chart of the HRP governance structure can be found in Annex 3. The governance of HRP is laid down in its Memorandum of Understanding, which defines the composition and role of PCC and STAG (see Chapter 1 Annex 2). PCC provides overall programme monitoring, strategic guidance and financial oversight, while STAG provides overall scientific and technical guidance on the Programme’s work. These are supplemented by two other important mechanisms, GAP and RP2. GAP advises the Programme on issues of gender equity and equality and human rights, and ensures a gender orientation in all its work. Reports of GAP are presented at both the STAG and the PCC meetings. RP2 reviews every research project proposed for support, and no project can be funded unless RP2 provides a positive scientific, technical and ethical review. 5.1 The Policy and Coordination Committee Membership of PCC is determined by the Memorandum of Understanding. All major policy, financial and strategic decisions are either made by or endorsed at PCC meetings. A clear valued-added mechanism for the governance of HRP is that the Programme maintains a system whereby it documents its follow-up actions to all recommendations made by PCC, and the subsequent meeting always provides the opportunity for PCC to examine and further review the Programme’s responses. (This mechanism also applies to STAG and GAP.) HRP External Evaluation 2008−2012 Chapter 3 Page | 83 PCC is made up of various categories of members. Category 1 members of PCC are the 11 largest donors to the Programme in the previous biennium; category 2 members are 12 representatives from programme countries selected by the WHO regional offices; and category 3 members are two additional members elected by a ballot of category 1 and 2 members every 3 years. Up to the present, category 3 members have always been countries, though the Memorandum of Understanding does not prohibit nongovernmental organizations or foundations from requesting to be considered. Category 3 membership has been raised in previous evaluations as a possible way of expanding PCC, but the number cannot be changed without modifying the Memorandum of Understanding. In reality, PCC discussions are very open. The chair recognizes anyone who wants to speak, including PCC observers, and almost all decisions are made by consensus rather than by vote. PCC is thus already a highly inclusive mechanism, and any changes to its current membership would most likely not result in any improvements in its effectiveness or efficiency. A continuing issue from previous evaluations of the Programme has been the participation of members from programme countries. The Programme itself has little say over either which countries are selected or the individual participants chosen to attend. The most appropriate attendees are probably national decision-makers on issues of sexual and reproductive health, in other words those individuals who will directly make use of the Programme’s products to change policies and programmes. Such persons would also be in the best position to review the overall strategies of the Programme and the relevance of its products for programme countries, and to provide feedback on how the Programme’s products have been used in their countries. A brief review of governance in the Special Programme for Research and Training in Tropical Diseases (TDR) revealed that it has recently constituted a subcommittee of its Joint Coordinating Board (JCB), its PCC equivalent, composed of cosponsors, and some donors and programme countries. The purpose of this committee is for the JCB to keep more in touch with TDR management in between JCB meetings, and to advise the director on any questions of policy that arise during the year. HRP has already moved in this direction by including the PCC chair in the meetings of the standing committee. Table 4 provides information on the views of respondents on the effectiveness of the PCC. The majority of respondents (more than 56%) gave a rating of “2” or better for the effectiveness of the PCC, but 80% of informants felt that the effectiveness of PCC could be improved. In elaborating their responses, respondents made a number of suggestions including: that more time should be allowed for discussion; that presentations should be shortened; that there should be more involvement of category 2 members; that the Programme should strengthen its reporting to PCC to include outcomes and impact as well as outputs; and that more time should be given to policy, strategic, and financial issues for PCC’s discussion, advice and agreement. A forward-looking example could be a PCC discussion on the positioning of HRP in the context of the MDGs and the International Conference on Population and Development (ICPD) beyond 2014. HRP External Evaluation 2008−2012 Chapter 3 Page | 84 Table 4 Respondents’ views on the effectiveness of the Policy and Coordination Committee Effectiveness Percentage of respondents 1 – highly effective 17.7 2 39.2 3 32.9 4 8.9 5 – not at all effective 1.3 Total (n = 79) 100.0 Respondents also felt that HRP should look at the modus operandi of the JCB of TDR for possible ways of improving the efficiency and effectiveness of PCC. Some informants indicated that sometimes PCC’s direct involvement in areas such as priority setting and communication was not appropriate, since the job of PCC was more to ensure that sufficiently robust priority- setting mechanisms and communication strategies were in place. Recommendations PCC needs to ensure that its agenda gives sufficient space for the discussion of policy, strategic and financial issues central to the well-being, growth and development of the Programme, as well as receiving reports on progress, outcomes and impact. PCC may wish to consider adding an agenda item every other year that would provide an opportunity for donors, cosponsors and programme countries to report on their use of the Programme’s products. 5.2 The Scientific and Technical Advisory Group As with PCC, STAG was also established in HRP’s Memorandum of Understanding (see Chapter 1 Annex 2). Its functions include the scientific and technical review of “the content, scope and dimensions of the Special Programme, including the research areas covered and approaches to be adopted, and to recommend priorities within the Special Programme”. As of 2012, the STAG had a full complement of 18 members, 10 of whom are female. The majority of members, 11, are from programme countries, and all WHO regions are well represented. In 1999, PCC endorsed the recommendation of STAG to expand its mandate to review the activities of RHR as a whole, rather than just HRP. This did not require any change to the Memorandum of Understanding, and it was determined that this additional work would be carried out by STAG members in a different capacity, without any financial implications for HRP, and was being done at the request of RHR. The cost of STAG, and also GAP, were to be paid in alternate years out of the budgets of PDRH and HRP. (However, because of core budget reductions in PDRH, the costs of STAG and GAP have been borne entirely by HRP since 2009.) In even-numbered years, STAG reviews cross-cutting strategic issues; in odd-numbered years, it reviews in-depth progress by HRP and PDRH teams over the last 2-year period, as well as their draft programmes of work for the next biennium. These reviews take into account the MDGs and other relevant global initiatives such as the UN Secretary-General’s Global strategy for women’s and children’s health (1). As can be seen from Table 5, more than 72% of respondents gave the STAG a rating of “1” or “2” on the effectiveness of its work. However, over 80% of respondents said that the effectiveness of STAG could be improved, and then elaborated their answers, as discussed next. HRP External Evaluation 2008−2012 Chapter 3 Page | 85 Table 5 Respondents’ views on the effectiveness of the Scientific and Technical Advisory Group Effectiveness Percentage of respondents 1 – highly effective 21.2 2 51.8 3 21.2 4 4.7 5 – not at all effective 1.2 Total (n = 85) 100.1 a This percentage over 100% is due to rounding. A number of respondents felt that STAG should take on a stronger strategic role in shaping the Programme’s priorities, and that this could be done, for example, by proposing reorientation or rebalancing of the research agenda in terms of the amounts of basic, social determinants, and implementation research; and providing guidance on priorities within programme components. A number of respondents felt that more communication with PCC in between annual meetings would increase the effectiveness of STAG, and that longer time for the discussion of STAG reports at the PCC meetings would also enable STAG to provide more strategic input. A number of respondents felt that STAG agendas were too crowded, allowing insufficient time for in-depth discussions, and that periodic in-depth reviews of each area of the Programme’s work by a subgroup of STAG, with one or two co-opted experts in the area, would be more fruitful and would put to better use the expertise that STAG contains. Such in-depth reviews would enable STAG to provide better guidance to the Programme in a particular area of work. Respondents also felt that STAG meetings often resulted in rather long “to do” lists, which put an additional burden on programme staff in following them up. A review of STAG meetings over the last 5 years revealed over 174 recommendations, and an additional number of suggestions and requests – an average of 35 per meeting. Almost all these recommendations urged action by the Programme, and these were often to increase work in certain areas, without identifying additional funds or areas to drop. Only one recommendation to discontinue work in a specific area was identified. STAG also engaged in a number of strategic discussions over the period 2008–2012, for example, on how the Programme could best assist countries in implementing the new MDG reproductive health access target; how the Programme could strengthen linkages between research on sexual and reproductive health and research on HIV/AIDS; and how it could address the issue of violence against women. However, the extent to which these discussions subsequently helped guide or modify the Programme’s workplan was not always clear. Some recommendations resulting from these discussions were rather broad, for example, “explore a diverse health-systems approach including the private sector; examine inequities to services amongst population subgroups; mobilize the H4 initiative2; and, pay more attention to adolescents”, and did not always seem to provide sufficient guidance on exactly what the Programme was expected to do. However, a strategic discussion on implementation research 2 UNFPA, UNICEF, WHO, the World Bank, UNAIDS, have joined forces as Health 4+ (H4+) to support countries with the highest rates of maternal and newborn mortality. The H4+ partners support emergency obstetric and neonatal care needs assessments and help to cost national maternal, newborn and child health plans, mobilize resources, increase the number of skilled birth workers, and improve access to reproductive health services. HRP External Evaluation 2008−2012 Chapter 3 Page | 86 did seem to provide an opportunity for HRP to help bridge the gap between research and action, and to strengthen its work in introducing its products into countries. Conclusion The guidance and advice provided to the Programme by STAG continues to be appreciated and viewed as very important. Since 2000, STAG has substantially increased its workload. Its meetings already lasted 4 days, so there was little margin for compensation here. The anticipated benefit of reviewing the totality of RHR’s work has, perhaps somewhat inevitably, led to less detailed STAG reviews of some aspects of HRP’s work. Since 2009, PDRH flexible funds have been used entirely to pay for staff salaries, thus further bringing into question the added value of STAG’s review of PDRH activities, which are largely driven by specified funds from various donors. Recommendation PCC may wish to consider a number of different options including the following: STAG could revert to its original function as the scientific and technical review body for HRP, and could receive and review a report only on the overall work of PDRH on a biennial basis; STAG could undertake in-depth reviews, perhaps in alternate years, of two to three of the main areas of the Programme’s work, using contribution to achievement of global goals such as MDG targets as one of the major indicators of the relevance of the work; in other years, it could focus on more strategic, policy and forward-looking issues, as well as reviewing and advising on overall workplans and budgets. 5.3 The Gender and Rights Advisory Panel The Gender Advisory Panel (GAP) was established in 1995, and renamed the Gender and Rights Advisory Panel in 2007. Its terms of reference include reviewing all aspects of the department’s work, with attention to gender equity and equality, as well as human rights; proposing mechanisms through which gender and rights concerns can be brought to bear across the department; and examining and giving guidance to the department on key concepts currently under debate in the area of sexual and reproductive health. As of 2012, GAP had eight members, from five of the six WHO regions, two of whom are male. Currently, the WHO Regional Office for South-East Asia is not represented as a member, though two temporary advisers from the region attended GAP in 2012. Three members come from donor countries and five from programme countries. Table 6 provides the views of respondents on the effectiveness of GAP. As for STAG, more than two thirds of respondents gave the effectiveness of GAP a rating of “1” or “2”. Eight out of ten respondents felt that the efficiency and effectiveness of GAP could be improved, and elaborated further on their comments, as discussed next. Table 6 Respondents’ views on the effectiveness of the Gender and Rights Advisory Panel Effectiveness Percentage of respondents 1 – highly effective 25.6 2 47.4 3 25.6 4 1.3 5 – not at all effective 0.0 Total (n = 78) 99.9 a This percentage under 100% is due to rounding. HRP External Evaluation 2008−2012 Chapter 3 Page | 87 Some respondents felt that GAP operated somewhat in isolation and needed to find a way of bringing its concerns more into the mainstream of HRP; other respondents felt that STAG has a degree of gender and rights expertise, and that there was thus the potential for some duplication. A number of respondents felt that GAP should be congratulated for having largely achieved its original goal of mainstreaming gender and rights into the work of HRP. Some respondents discussed the need for communication among members in between meetings, perhaps by an e-forum, and also that the GAP report should be made available earlier before PCC meetings, and given more time for discussion during PCC meetings. As with STAG, GAP issues a large number of recommendations, averaging around 30 per meeting. Some informants felt that these needed to be better prioritized in order to have a greater impact, and in order not to unnecessarily over-burden staff in following them up. Conclusion The guidance and advice provided to the Programme by GAP continues to be appreciated, and GAP continues to be viewed as having a very important role to play. GAP is perceived as having largely achieved the objective of mainstreaming gender and rights issues into HRP’s work. It is also perceived as remaining somewhat isolated from the mainstream of HRP. A proportion of its meetings cover similar agenda items to STAG, and part of the function of GAP is to provide a gender review of the Programme’s work. A simultaneous scientific, technical and gender review of HRP’s work would enrich and add value to the process of providing guidance to the Programme, and, at the same time, further reduce direct and indirect governance costs. Recommendation HRP should examine the feasibility of merging GAP into STAG. This would require ensuring that STAG maintains adequate gender and sexual and reproductive health rights expertise; carries out biennial reviews of HRP’s full programme of work from a gender and rights perspective; and commissions an independent review of its approach to gender and human rights after 5 years. 5.4 The Research Project Review Panel RP2 is tasked with providing a technical, scientific, financial and ethical review for every research project the Programme proposes to undertake. No research proposal can receive funding from HRP without final approval following RP2 review. RP2 is a vital mechanism and resource for the Programme, in ensuring the quality and integrity of its research. It is made up of more than 50 worldwide experts in disciplines that cover the range of HRP’s work. The members of RP2 work pro bono, and other mechanisms prevent any real or perceived conflicts of interest. Since 2008, a number of changes have been made in the project review process. Prior to the formation of RP2, research proposals were subject to four levels of review, which inevitably created delays in their approval. As a result of a review of this process, the five specialist panels (social science research, basic and biomedical research, epidemiological research, toxicology, and country programme development) were integrated with the Scientific and Ethics Review Group (SERG), essentially coalescing six panels into one new mechanism – RP2. RP2 reviews around 100 research proposals per year, but also continues to perform the previous functions of the specialist panels, such as reviewing concept notes and advising on preliminary proposals to assist in their further development. The creation of RP2 has resulted in a review process that is simpler but retains its original rigor; a process that has dramatically reduced, since 2008, the average time for review and approval from 18 months to 3 months; an updated, simplified electronic HRP research application form, HRP External Evaluation 2008−2012 Chapter 3 Page | 88 which replaced the previous hard-copy “Orange Book”, together with revised guidance documents; and preparation of new ethical guidance for social science and implementation research. Detailed terms of reference and rules of procedure were prepared for RP2 and, in addition, it continues to carry out ethics-capacity-strengthening activities in a number of countries each year, though there has been a decrease in these activities since late 2011, owing to budgetary constraints. Since May 2010, more than 575 researchers from programme countries have received training or updating in the ethical principles of research. The creation of RP2 has resulted in an estimated biennial saving in excess of US$ 250 000 in terms of meeting costs. It has also resulted in considerable staff savings. Previously, six professional and six administrative support staff members serviced the five specialist panels and SERG on a part-time basis; currently, one professional and one administrative support staff member service RP2 on a part-time basis. These additional savings are estimated at somewhere between US$ 450 000 and US$ 500 000 each biennium. Despite the rigorous RP2 review process, all HRP research projects also have to be submitted to and approved by the WHO Research Ethics Review Committee (ERC), though the value added of this additional procedure is somewhat in doubt, and it appears to be a process of duplication. The ERC has, on occasion, gone beyond its terms of reference and made technical comments on proposals, which then require further responses, adding to the approval time. More importantly, as of this time, no proposal submitted by RP2 to ERC has been turned down on ethical grounds. Members of RP2 have also, at the request of ERC, provided ethics-capacity- strengthening training to ERC members. ERC is financed through appropriations from WHO research programmes such as HRP. The Programme is thus essentially required to pay twice for ethical reviews of its research, and over the last 3 years, these costs have amounted to US$ 60 000 in 2010, US$ 107 000 in 2011 and US$ 65 300 in 2012. These figures do not include the hidden costs of the extra time spent by professional and administrative staff, as well as country investigators, as a result of this additional process. Thus, the savings in meeting costs made by the Programme in streamlining its scientific, technical and ethical review process (US$ 250 000 in the 2010–2011 biennium) were offset by a subvention of US$ 167 000 to ERC during the same period. Table 7 provides information on the views of respondents on the effectiveness of RP2. Just over 67% of respondents who gave a specific response rated RP2 as “1” or “2” on the scale of effectiveness, over 90% gave a rating of “3” or better. Again, over 80 per cent of informants felt that RP2 could be improved in some way, and in elaborating on their responses indicated the following: many felt that timeliness of the review process had been improved, but that proposals sometimes needed more prescreening, review and revision by HRP staff before being submitted; some felt it had been a mistake to do away with the specialist review panels, though they acknowledged the considerable cost savings of the new process, and others thought that the Programme may have lost some capacity in scientific research direction and monitoring oversight with the loss of the strategic committees; a number felt that the ERC process was duplicative, and costly in terms of time and money. HRP External Evaluation 2008−2012 Chapter 3 Page | 89 Table 7 The Effectiveness of the Research Project review Panel in RP2 in providing scientific and ethical review of HRP’s research projects Effectiveness Percentage of respondents 1 – highly effective 23.4 2 43.8 3 23.4 4 6.3 5 – not at all effective 3.1 Total (n = 64) 100.0 Conclusion RP2 performs an essential function in the technical, scientific, financial and ethical review and approval of HRP’s research proposals. The creation of RP2 has strengthened and harmonized the research-proposal review process, and is significantly more efficient, in terms of time and money, than the multilevel review process it replaced. The ERC ethical review essentially duplicates the ethical review already carried out by RP2 and costs HRP additional time and money. One of the more common reasons for delays in the approval process is the submission to RP2 of research proposals that are incomplete, or that do not meet the required scientific and technical standards. Recommendations The Programme should renegotiate its relationship with regard to the overlapping functions that exist between RP2 and ERC. Ideally, a way needs to be found for WHO senior management to entrust the ethical review of HRP’s research to RP2. This will most likely require a number of actions, including investment in a more robust RP2 database with support for data management, and application by RP2 for FWA-OHRP accreditation (Federal Wide Assurance for the Protection of Human Subjects – Office for Human Research Protections Database), which would include a system of periodic external reviews of RP2. When submitting research proposals to RP2 for final assessment and approval, programme staff should ensure that the proposals are complete and conform to the required technical and scientific standards. In addition to the regular annual review of ongoing research proposals, programme staff should consult RP2 at any point after a research proposal has been approved, if any scientific, technical, ethical or management issues arise during the lifetime of the project until its completion. For its major areas of work, the Programme needs to develop mechanisms for identifying research needs and priorities, as well as planning and monitoring research studies, utilizing external expertise. 5.5 Costs of HRP governance Table 8 shows the direct costs of HRP governance mechanisms for three time periods, 2002– 2003, 2006–2007, and 2010–2011. The overall total costs of governance have decreased considerably, by 33.6% between 2002–200 and 2010–2011, and by 22.5% over the period of the current evaluation. Furthermore, these lower direct costs do not reflect the additional savings attributable to the reduction in HRP staff time spent contributing to, supporting and attending the reduced number of meetings that now take place. HRP External Evaluation 2008−2012 Chapter 3 Page | 90 Table 8 Direct costs of HRP governance, 2002 – 2011, US$ Costs, US$ 2002–2003 2006–2007 2010–2011 Policy and Coordination Committee 176 359 222 159 267 686 Scientific and Technical Advisory Group 187 139 140 352 246 168 Gender Advisory Panel 81 509 54 842 126 802 Standing committee 7385 3569 10 074 Regional advisory panel, Region of the Americas 37 795 137 717 34 222 Regional advisory panel, South-East Asia and Western Pacific Regions 70 416 95 467 7302 Regional advisory panel, African and Eastern Mediterranean Regions 86 943 104 934 38 725 Regional advisory panel, European Region 53 759 — — Strategic Committee on Promoting Family Planning 48 807 — — Strategic Committee on Making Pregnancy Safer 46 476 — — Strategic Committee on Addressing Sexually Transmitted Infections 50 351 — — Strategic Committee on Preventing Unsafe Abortion 53 364 — — Scientific and Ethical Review Group 222 930 262 560 7462 Special Panel on Social Science Research 106 754 122 617 77 570 Special Panel on Basic and Biomedical Research 86 826 14 978 — Special Panel on Epidemiological Research 115 651 7943 — Special Panel on Country Programme Development — 59 783 — Research Review Panel — — 135 113 Total 1 432 464 1 226 921 951 124 Between 2002–2003 and 2006–2007, the reduction in costs of over US$ 200 000 was largely due to the discontinuation of the four strategic committees on promoting family planning, making pregnancy safer, addressing sexually transmitted infections, and preventing unsafe abortion. Between 2006–2007 and 2010–2011, the reduction in costs of more than US$ 275 000 was largely due to the discontinuation of the specialist panels and SERG, and their replacement by RP2. This change resulted not only in cost savings but also in greater efficiency, owing to a more predictable and stable schedule of meetings, and thus faster overall review of HRP research proposals. The costs of RAPs have also been reduced considerably, mainly through greater use of video- and/or teleconferencing. Over the period between 2002–2003 and 2010–2011, the costs of PCC, STAG and GAP have increased 51.8%, 31.5% and 55.6% respectively, largely owing to increases in the costs of travel and accommodation. HRP External Evaluation 2008−2012 Chapter 3 Page | 91 During the period 2008–2012, one meeting of PCC was held outside Geneva in a programme country, which allowed PCC members to visit the sites of several research projects and to see first-hand the results of HRP’s work. While this had clear advantages in terms of underscoring the relevance of HRP’s work and putting PCC in context, it also had a significant financial implication, increasing the cost of the meeting by a factor of four, as compared to PCCs held in Geneva. Conclusion The Programme should be commended on continuing to carefully manage, and indeed reduce, its governance costs, and there remain few opportunities for further savings. An eventual merging of GAP with STAG would bring some additional savings and a further reduction in servicing costs. Recommendation The Programme should consider periodically holding a PCC meeting outside Geneva, but only after prenegotiating a cost-sharing agreement with the host government. 6 Management and administration 6.1 Managing for results Did HRP achieve its objectives as laid down in its biennial workplans? HRP’s workplan is developed within the overall context of the WHO workplan and budget, in order to show how the work of the Programme relates and contributes to the achievement of WHO’s overall expected results. HRP’s work contributes to WHO Strategic Objective 4, “To reduce morbidity and mortality and improve health during key stages of life, including pregnancy, childbirth, the neonatal period, childhood and adolescence, and improve sexual and reproductive health and promote active and healthy ageing for all individuals”, and specifically to WHO Organization-Wide Expected Result (OWER) 4.2, “National research capacity strengthened as necessary and new evidence, products, technologies, interventions, and delivery approaches of global and/or national relevance available to improve maternal, newborn, child and adolescent health, to promote active and healthy ageing, and to improve sexual and reproductive health”. HRP’s expected results are clearly identified in its biennial workplans. These are largely quantitative, in terms of, for example, numbers of studies to be carried out and numbers of reports to be published. For example, in its 2010–2011 programme budget, HRP identified a number of indicators and targets for achievement during the biennium. At the end of the biennium, as a part of WHO reporting, the actual achievement values were collected, and these, together with the targets and indicators are shown in Table 9. HRP External Evaluation 2008−2012 Chapter 3 Page | 92 Table 9 HRP indicators, targets and achievement values for 2010–2011 Indicator Target Achievement 1.1 Number of completed studies with new or updated evidence 57 40 1.2 Number of new or updated systematic reviews on best practices, policies and standards of care in sexual and reproductive health 32 39 1.3 Number of new or updated evidence-based guidelines for sexual and reproductive health programmes and services 17 8 1.4 Number of new or updated evidence-based tools or briefs for sexual and reproductive health programmes and services 20 16 1.5 Number of externally published, peer-reviewed papers 66 87 1.6 Number of new partners entering formal collaboration for sexual and reproductive health 18 26 1.7 Number of centres (new) strengthened to conduct research to contribute to the achievement of universal access to sexual and reproductive health 9 10 1.8 Number of training activities and grants provided to promote sexual and reproductive health 44 77 2.1 Events to introduce WHO guidelines and tools for sexual and reproductive health services and programmes 38 36 2.2 Countries (new) supported to test WHO guidelines on sexual and reproductive health 20 5 2.3 Countries (new) supported to implement WHO guidelines on sexual and reproductive health 17 21 2.4 Countries (new) supported to test WHO strategies for sexual and reproductive health 11 11 2.5 Countries (new) supported to implement WHO strategies for sexual and reproductive health 31 32 2.6 Countries implementing interventions based on research conducted or supported by RHR/HRP to improve sexual and reproductive health 13 21 3.1 Advocacy events/initiatives supported to promote universal access to SRH 59 44 As can be seen from Table 9, in some areas, such as published research papers, training activities and grants, and countries implementing interventions based on HRP research, the Programme far exceeded its targets. In other areas, such as evidence-based studies and tools, and countries supported to test HRP’s sexual and reproductive health guidelines, achievement was less than expected. Overall, the Programme clearly exceeded the targets for its quantitative indicators. However, such reporting captures neither the potential nor the actual impact of the Programme’s work, and, in reality, HRP does not have the funds, capacity or mandate to ensure that its products are used. HRP has some control over potential impact, by ensuring that its work focuses on those areas and questions that are most relevant in assisting programme countries to improve the sexual and reproductive health of their populations, and it does have the mandate to undertake research on the introduction of its products, and research on barriers to the uptake of its products. This is essential in demonstrating to its clients and constituents both the value of its products and how they can be scaled up for use in countries, so that the much larger funds available to multilateral, bilateral and other donors can be guided towards being used for interventions that HRP has shown are evidence based, and thus interventions with an almost “guaranteed” impact. Two thirds of respondents gave a rating of “1” or “2” for the Programme’s effectiveness in setting its workplan targets, and a further 30% gave a rating of “3”. HRP External Evaluation 2008−2012 Chapter 3 Page | 93 Respondents’ views on HRP’s effectiveness in monitoring its workplan targets are given in Table 10. The majority of respondents gave positive ratings, 65.4% giving a rating of “1” or “2”, and a further 30.8% gave a rating of “3”. Some respondents felt that the Programme should strengthen its reporting of results to PCC, providing more data on outcomes, and more information on the impact in programme countries, particularly examples of implementation research and research to overcome barriers to introducing evidence-based policies and programming. Table 10 Respondents’ views on the effectiveness of HRP in monitoring its workplan targets Effectiveness Percentage of respondents 1 – highly effective 19.2 2 46.2 3 30.8 4 1.9 5 – not at all effective 1.9 Total (n = 52) 100.0 Conclusion Respondents gave a very positive assessment of HRP’s effectiveness in managing for results. The evaluation took note of the dual reporting requirements for the HRP secretariat. On the one hand, HRP is required to condense its entire programme of work under three indicators, for the WHO expected result OWER 4.2, while on the other hand, its cosponsors, donors and governing and advisory bodies reasonably expect a far more comprehensive results framework. For the current 2012–2013 biennium, the Programme is continuing to use the same framework. Recommendation For future biennia, starting in 2014–2015, HRP should develop a new results framework which, in addition to a simplified approach to counting outputs, should identify and monitor utilization of its products in programme countries, and, wherever possible, identify potential and/or actual impact. 6.2 Managing research grants How effective is HRP in awarding, processing and monitoring research grants? Table 11 gives respondents’ views on the issue of research grants. The majority of respondents gave positive ratings for HRP’s effectiveness in this area. More than three quarters of respondents gave a rating of “3” or better in all categories. In elaborating on their responses, some respondents noted that the awarding of grants was sometimes slow, but that time was needed for the necessary scientific, technical and ethical review. Others again noted the need for HRP staff to prescreen and make sure that proposals were sufficiently strong, particularly from a scientific and technical point of view, before being submitted to RP2, and that this would help speed up the awarding of grants. Income flows were also noted as sometimes affecting the Programme’s ability to award and process grants. HRP External Evaluation 2008−2012 Chapter 3 Page | 94 HRP’s effectiveness in awarding, processing and monitoring research grants Action category Percentage of informants giving a rating of “1” (highly effective), “2” or “3” Awarding grants 78.7 Processing grants 84.0 Monitoring grants 77.1 In response to the question: “How have changes in WHO administrative processes affected the processing of research grants? (1: very positively to 5: very negatively)”, less than 20% of respondents gave a rating of “1” or “2”, and more than 80% gave a rating of “3” or worse. Reasons cited included staff reductions; the additional requirement for passing proposals through the WHO ERC, which often raised additional questions and discussions that were rarely crucial, sometimes redundant, and only increased processing time; and communication and processing difficulties with the new WHO administration unit in Kuala Lumpur, although some problems, such as the ability to continue processing advance payments, have been subsequently resolved. The somewhat lower ratings for monitoring were mirrored in the more detailed responses of respondents, who cited various reasons, including over-burdened programme staff, lack of resources, and unclear links with individual performance-assessment mechanisms, which needed strengthening and linking to clear deliverables. Conclusion The Programme is viewed as performing efficiently the processing, awarding and monitoring of research grants. 6.3 Managing research Was the Programme implemented in the most efficient way? The questionnaire also examined the issue of the balance between work being undertaken by programme staff, and work contracted out. Changes here could improve both effectiveness and efficiency. A total of 46.5% of respondents felt that the balance between research work undertaken by programme staff, and research work commissioned by HRP but contracted out, was about right. However, an almost equal proportion, 45.3%, felt that the balance should shift towards work contracted out. Less than 10% of respondents felt that the balance should be towards more work undertaken by staff. In elaborating on their responses, a number of informants felt that while it was essential for HRP to continue to set the research agenda, it should use more institutions, particularly in programme countries, to carry out the research. The Programme should continue to set the agenda, and programme staff should continue to guide the development of research proposals, help train investigators in research methodology, monitor research in progress, and monitor data processing and analysis and the writing up of the results. These respondents felt that a greater proportion of research, systematic reviews and methodological work could be implemented by institutions outside WHO, and would thus enable the Programme to manage a larger portfolio of work with fewer staff, and thus to become more efficient. Many respondents related their answers to this question to the issue of HRP staffing costs, which represent a major proportion of overall programme costs. A brief review of other programmes in WHO revealed that such costs vary considerably. For example, staffing costs HRP External Evaluation 2008−2012 Chapter 3 Page | 95 for TDR and the Global Alliance are, respectively, around 20% and 30% of their overall budgets, far lower than in HRP, where these costs are set at a maximum of 40% of approved budget. Some respondents felt that the changes in the functions of the statistical and data-processing group in HRP has made one of their key functions the “prequalification” of institutions/organizations to which data management and processing could be outsourced – in other words, to manage and quality control data processing rather than doing it, except it situations of “force majeur”. This both improves the efficiency of HRP and further strengthens capacity in programme countries. Conclusion The proportion of income spent on staff costs in HRP is considerably greater than for other research programmes in WHO. The proportion of research and other work being undertaken directly by programme staff appear to be high. Over time, the function of the statistical and data-processing group in HRP has changed from doing the work to quality controlling the data- processing work being done by outside institutions. Recommendation In order to gain further efficiencies, the Programme may need to re-examine the balance between the proportion of research being done by programme staff and the proportion being managed by programme staff but implemented by outside institutions. HRP External Evaluation 2008−2012 Chapter 3 Page | 96 Reference 1. United Nations Secretary-General. Global strategy for women’s and children’s health. New York, United Nations, 2010 (http://www.who.int/pmnch/topics/maternal/201009_globalstrategy_wch/en/index.h tml, accessed 19 December 2012). Chapter 3 Page | 97 Annex 1: Trends in HRP income from royalties, 2000–2011 Year 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Total income, US$ 17 968 000 15 297 000 14 377 000 13 031 000 13 324 000 11 830 000 23 371 000 13 715 000 20 915 000 19 810 000 18 941 000 22 728 000 Royalties, US$ Update Software Ltd — — 3070 1070 364 1274 1081 — — — — — Richter Gedeona — — — — — 80 389 100 191 233 849 281 120 423 148 553 310 564 028 Schering 304 300 410 915 318 661 96 293 42 294 John Wiley & Sons — — — — 276 — — — — — — — Women’s Capital Corporation 50 000 100 000 100 000 — — — — — — — — — Total royalties, US$ 354 300 510 915 421 731 97 363 42 934 81 663 101 272 233 849 281 120 423 148 553 310 564 028 Royalties as a percentage of total income 1.97 3.34 2.93 0.75 0.32 0.69 0.43 1.71 1.34 2.14 2.92 2.48 Chapter 3 Page | 98 Annex 2: Summary of HRP resource-mobilization activities 2008–2012 2008 2009 2010 2011 2012 Donor roundtables1 None 1 – Friends of HRP, RHR hosted WHO HQ 1 – Hosted by Netherlands, Geneva Mission 1 – Friends of ICPD: Norway/Netherlands 0 Marketing and outreach materials published 5 – Theme initiative partner briefs 6 – Thematic partner briefs Approximately 10 Approximately 17 Estimated 2 E-marketing RHR/HRP news (e-mail partner newsletter) RHR/HRP news (e-mail partner newsletter) RHR/HRP news (e-mail partner newsletter) RHR/HRP news, reproductive health update (public e-mail shot) RHR/HRP news, reproductive health update End-of-year director letter Paul Van Look Paul Van Look Mike Mbizvo Mike Mbizvo Planned Christmas cards and handwritten letters Send by professional staff – approximately 200 Send by professional staff – approximately 250 Send by professional staff – approximately 350 Send by professional staff – about 400 Planned HRP brochures High-quality HRP brochure 0 0 0 2 high-quality HRP brochures Resource-mobilization strategy Updated annually Updated annually Updated annually Updated annually Updated annually Resource-mobilization training Informally upon request Resource-mobilization training provided for the Regional Office for the Eastern Mediterranean Informally upon request Informally upon request Informally upon request New donors Member States None None None None 1 – INCa (France) Foundations, civil society, other 1 – James Tudor Foundation 1 – University of North Carolina 1 – Gynuity 3 – Implementation Research Platform; Save the Children; New Venture Fund 1 – Anonymous Multilateral/cosponsors 0 0 0 1 – Multidonor trust fund: sexual violence 1 – UNICEF New donors total 1 1 1 3 3 Returning donors Member States 0 1 – Italy 1 – Japan 0 0 Foundations, civil society, other 0 2 – FHI; OSI 0 3 – Bill and Melinda Gates Foundation; European Commission; Wellcome Trust 1 – UNFIP/UNF 1 Roundtables are informal invitation-only briefings provided for donors that are normally cohosted between WHO and a lead partner. Chapter 3 Page | 99 2008 2009 2010 2011 2012 Multilateral/cosponsor 0 2 – UNICEF; UNDP 0 0 0 Returning donors total 0 5 1 3 1 Donors with an increase that year maintained Member States 0 1 – Flanders 0 4 4 – Flanders; Norway; USA Netherlands Foundations, civil society, others 0 1 – Anonymous 1 – Packard Foundation 1 – Bill and Melinda Gates Foundation 0 United Nations 0 1 – UNFPA 0 1 0 Donors with a contribution increase total 0 3 1 6 4 FHI, Family Health International; INCa, Institut National du Cancer (French National Cancer Institute); OSI, Open Society Institute; UNF, United Nations Fund; UNFIP, United Nations Fund for International Partnerships. HRP External Evaluation 2008−2012 Chapter 3 Page | 100 Annex 3: HRP governance structure Page | 101 Chapter 4 Evidence generation and synthesis to improve family planning, prevent unsafe abortion and prevent and control sexually transmitted and reproductive tract infections Affette McCaw-Binns Professor of Reproductive Health and Epidemiology and Head, Department of Community Health and Psychiatry, University of the West Indies, Jamaica Jasneth Mullings Research Fellow, Department of Community Health and Psychiatry, University of the West Indies, Jamaica Page | 102 Page | 103 Contents Abstract 107 Introduction 107 Methods 107 Results 107 Conclusions and recommendations 114 Summary 116 1 Introduction 117 2 Methods 117 2.1 Phase I: fact finding with WHO programme teams 118 2.2 Phase II: desk review and consultation with stakeholders 118 3 Rationale 118 4 Findings 119 4.1 General 119 4.2 Improving family planning: generating new evidence, marketing new products and protecting health and safety 120 5 Conclusions and recommendations 141 5.1 Summary 141 5.2 Impact 141 5.3 Recommendations 143 References 144 Annex 1: List of persons interviewed and methods of contact 148 Annex 2: Revised guidance: combined hormonal contraceptive use during the postpartum period 149 Recommendation 149 Annex 3: GRADE evidence profiles – systematic reviews of hormonal contraception use and HIV risk 150 A. GRADE evidence profile: hormonal contraception for women at high risk of HIV 150 B. GRADE evidence profile: risk of HIV transmission among women living with HIV and using hormonal contraceptives 151 C. GRADE evidence profile: HIV disease progression among women living with HIV and using hormonal contraception 152 Page | 104 Abbreviations ANC antenatal care ART antiretroviral therapy BMA British Medical Association CAC comprehensive abortion care CERRUGUI Cellule de Recherche en Santé de la Reproduction en Guinée (NGO in Guinea) CCUP comprehensive care for unwanted pregnancy CHC combined hormonal contraceptive COC combined oral contraceptive COPUA Coalition for Prevention of Unsafe Abortion DAC Development Assistance Committee (OECD) DMPA depot medroxyprogesterone acetate (an injectable hormonal contraceptive) DSMC data safety and monitoring committee EC emergency contraception ECS elimination of congenital syphilis ERC Ethical Review Committee (WHO) EVA electronic vacuum aspiration FP family planning FSH follicle-stimulating hormone GRADE grading of recommendations, assessment, development and evaluation (framework for reviewing evidence) HRP UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction ICEC International Consortium for Emergency Contraception IEC information, education and counselling IUD intrauterine device IUD intrauterine device LAC Latin America and the Caribbean LH luteinizing hormone LMP last menstrual period MDG Millennium Development Goal MEC Medical eligibility for contraceptive use (WHO: 4th edition 2009) MR menstrual regulation MTCT mother-to-child transmission MVA manual vacuum aspiration NET-EN norethisterone enantate (an injectable progestin used as a contraceptive) Page | 105 NGO nongovernmental organization OCP oral contraceptives pill OECD Organisation for Economic Co-operation and Development OTC over the counter PAC post-abortion care PPTC prevention of parent-to-child transmission (of HIV) RHR WHO Department of Reproductive Health and Research RNA ribonucleic acid SRH sexual and reproductive health STI sexually transmitted infection TU testosterone undecanoate (an injectable androgen) UN United Nations UNAIDS Joint United Nations Programme on HIV/AIDS UNDP United Nations Development Programme UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund VTE venous thromboembolism WHO World Health Organization Page | 106 HRP External Evaluation 2008−2012 Chapter 4 Page | 107 Abstract Introduction The World Health Organization’s (WHO) Department of Reproductive Health and Research (RHR) is the implementing body for the multi-agency (United Nations Development Fund (UNDP)/United Nations Population Fund (UNFPA)/United Nations Children’s Fund (UNICEF)/ WHO/World Bank) Special Programme of Research, Development and Research Training in Human Reproduction (HRP). The 2008–2012 case-study highlights HRP activities in the areas of (a) family planning (FP), (b) unsafe abortion and (c) sexually transmitted infections (STIs), by demonstrating HRP’s unique process of addressing sexual and reproductive health (SRH) is- sues, from problem identification to generating new knowledge, to global roll-out of solutions. Methods Meetings were held with team leaders in Geneva (18–20 April and 19–22 June 2012), to identi- fy focal activities of the professional clusters. A desk audit of peer-reviewed publications and WHO reports was complemented by e-mail contact with collaborators. Highlighted work in- cludes the development of a male contraceptive, marketing of emergency contraceptives and HRP’s response to evidence about hormonal contraceptive use and HIV risk. The application of the WHO strategic approach to reducing unsafe abortion and the initiative to eliminate con- genital syphilis are also summarized. Results Family planning: developing a male contraceptive A phase II, eight-country trial of two long-acting injectable hormones, testosterone un- deconate (TU), an androgen, and norethisterone enantate (NET-EN), a progestin, was under- taken to determine their safety, effectiveness and acceptability as a male contraceptive. While the regimen was effective in suppressing spermatogenesis, higher than expected rates of mood changes and increased libido led to a decision to stop the trial in April 2012. All sites should complete “close-out” visits by November 2012 and reporting is expected in 2013. While further testing of this drug combination is not expected to continue, the study will inform fu- ture development of a male contraceptive. The side-effects apply to the 1000 mg TU/200 mg NET-EN combination, and not to the drugs used alone for their approved indications. Family planning: getting a new product to market – emergency contraception The emergency contraceptive levonorgestrel can prevent pregnancy if taken within 120 hours of unprotected sexual intercourse. As it cannot terminate an established pregnancy, it is also acceptable where abortion is not. It is safe for over-the-counter (OTC) use and HRP has pro- moted its distribution through social marketing. As of 2010, 126 of 189 countries worldwide have an emergency contraception (EC) product registered, without the need for a prescription in 103 of these countries (see Table 1). Successful programmes (based on high-volume sales) were supported and/or promoted by governments. OTC sale at affordable prices, with discrete access through educated shop owners, pharmacists and health-care providers, facilitated sales. Barriers included lack of public-sector support; negative campaigns suggesting that EC acts as an abortifacient; prescription requirement; and limited public education. HRP External Evaluation 2008−2012 Chapter 4 Page | 108 Table 1 Countries with access to emergency contraception, by availability and region Region Available without prescription (pharmacist/OTC) Prescription required North America Canada, Mexico United States of America (nine states no restriction)a Caribbean Antigua and Barbuda, Aruba, the Bahamas, Barbados, Belize, Cura- cao, the Dominican Republic, French Guiana, Haiti, Jamaica, Puerto Rico, Saint Lucia, Trinidad and Tobago Latin America Argentina, the Bolivarian Republic of Venezuela, Ecuador, El Salva- dor, Guatemala, Nicaragua, the Plurinational State of Bolivia, Brazil, Chile, Colombia, Cuba, Paraguay, Peru Western Europe Austria, Belgium, Denmark, Iceland, Finland,b France, Greece, Lux- embourg, the Netherlands, Norway, Portugal, Spain, Sweden, Swit- zerland, the United Kingdom of Great Britain and Northern Ireland Germany, Italy, Lithuania Eastern Europe Albania, Armenia, Azerbaijan, Belarus, Estonia, Kazakhstan, Kyrgy- stan, Latvia, Lithuania, Poland, Romania, Serbia, Slovakia, Slovenia, Tajikistan, Ukraine Bulgaria, Czech Republic, Georgia, Hungary, the Russian Federation Africa Algeria, Benin, Burkina Faso, Cameroon, the Congo, Côte d’Ivoire, the Democratic Republic of the Congo, Egypt, Ethiopia, Gabon, Ghana, Guinea, Kenya, Lesotho, Libya, Madagascar, Mali, Mauritius, Moroc- co, Namibia, the Niger, Nigeria, Senegal, South Africa, Togo, Tunisia, Uganda, the United Republic of Tanzania, Zambia Botswana Middle East Cyprus, the Islamic Republic of Iran, Israel, Mauritius, Saudi Arabia, Turkey, Yemen Lebanon Asia China, Hong Kong, India, Japan, the Lao People’s Democratic Repub- lic, South Korea, Sri Lanka, Thailand, Viet Nam Bangladesh, Indonesia, Malaysia, Myanmar, Pakistan, Singapore, Taiwan Oceania Australia, French Polynesia, New Zealand Total 103 23 a Some states restrict use for adolescents aged <18 years or <16 years. b Not for adolescents aged <15 years. Family planning: monitoring product safety – hormonal contraception and risk of HIV acquisition The WHO Medical eligibility criteria for contraceptive use (2009) indicate that hormonal con- traceptives are safe for use by women at high risk of HIV or living with HIV; however, emerging evidence of associations between hormonal contraceptive use and risk of HIV infection, dis- ease progression and transmission, suggested the need to review the current guidance. In ear- ly 2012, HRP convened a team of 75 experts to review the new evidence. They concluded that the available evidence did not establish a clear causal association between injectable contra- ceptives and HIV acquisition, but did not rule out a possible effect; therefore, use of hormonal contraceptives should remain unrestricted but further evidence should be closely monitored. Women at high risk of HIV infection can use all existing hormonal contraceptive methods with- out restriction, but should use condoms and other measures to prevent and reduce their risk of HIV/STIs. Women living with HIV infection can also use all hormonal contraceptive methods without restriction but should consistently use condoms to prevent HIV transmission to non- infected sexual partners. Safe abortion services: application of the WHO strategic approach to unsafe abortion – experience in sub-Saharan Africa, eastern Europe and Asia Worldwide, 55% of abortions are classified as unsafe, more so in Africa (97%). In eastern Eu- rope, where abortion is legal, limited access to contraceptives results in overuse of abortion, with excess morbidity and mortality. In 2011, Bangladesh requested WHO assistance to im- HRP External Evaluation 2008−2012 Chapter 4 Page | 109 prove the quality of their menstrual regulation (MR)1 programme. The WHO strategic ap- proach is a useful tool for tackling sensitive problems and has been applied to the prevention of unsafe abortion. The experiences of 11 countries that applied the approach to address un- safe abortion during 2008–2012 are summarized. Stage 1: national strategic assessments Country assessment teams administered qualitative and quantitative interviews to a range of stakeholders and presented their findings at national dissemination workshops, where rec- ommendations were reviewed and refined and interventions prioritized. Common barriers to safe abortion care across all settings included the lack of standards and guidelines for compre- hensive abortion care and inadequate training of abortion providers. Some African countries that had ratified the 2003 Maputo Protocol on safe abortion reported that laws were in con- flict with this protocol. Where laws had been revised, the public was generally unaware of the changes. Other challenges included gender inequality; limited access to education for girls; and girls being expelled from school when they become pregnant. Stages 2 and 3: country-specific follow-up plans and activities Stage 2 activities in African countries included the development of national guidelines for abortion care; dissemination of information on legal indications for abortion; and strengthen- ing of FP programmes, adolescent SRH services and sexuality education. In the Republic of Moldova, a comprehensive abortion care programme was piloted in three perinatal centres, which were later designated as training sites for the stage 3 roll-out. In 2011, the WHO Safe abortion: technical and policy guidance for health systems was adapted for use in Bangladesh, with HRP assistance. The nature and pace of stage 3 scale-up depends on a health system’s readiness for change and available resources. Resistance to change has been reported where institutions or individuals benefited from the status quo. Lessons learnt The participatory process of the strategic approach helps catalyse change through early en- gagement of stakeholders. Ministry of health participation is essential to enable the integra- tion of recommendations into health policy and programmes. The process helps build local skills in planning, quantitative and qualitative data collection and analysis, intersectoral collab- oration, evaluation and implementation. Engaging financial partners early is critical, so that implementation does not lose momentum. Each country adapted the strategic assessment tool to the local context, with each interven- tion providing lessons for other country teams seeking ideas of where to begin. The methodol- ogy of the strategic approach is a powerful tool for addressing sensitive problems such as un- safe abortion and has robust applicability across a diverse range of settings. Controlling sexually transmitted and reproductive tract infections: initiative to elimi- nate mother-to-child transmission of congenital syphilis Untreated syphilis in pregnancy can cause late abortion, stillbirth, prematurity/low birth weight, neonatal deaths and congenital infection. These adverse outcomes may be avoided by antenatal screening and treatment. In 2007, WHO launched the global initiative to eliminate congenital syphilis by at least 80% in 10 high-burden countries (see Table 2) by 2015. Specific 1 Defined as “a procedure to make the menstrual cycle regular if menstruation is absent for a short duration” (Na- tional menstrual regulation services guidelines. Dhaka, Ministry of Health and Family Welfare, People’s Republic of Bangladesh, 2011). HRP External Evaluation 2008−2012 Chapter 4 Page | 110 targets are that by 2015, at least 90% of pregnant women will be screened for syphilis and at least 90% of women who are syphilis seropositive will be appropriately treated. Table 2 Countries applying the WHO strategic approach to safe abortion, by stage of the process, 2012 WHO Region Country Initial year Strategic stage in 2011/2012 Objective/output/outcome Partners African Region Ghana 2005 Research/pilot programmes under way; scale-up initiated Develop national standards and guidelines Ipas; HRP Guinea 2009 Dissemination workshop 2010; no further progress Improve access to legal abortion; inform a national effort to reposition FP HRP; CERRU- GUIa Malawi 2009 Research/pilot programmes under way (provision of DMPA by com- munity health workers) Revise sexuality education curricula; strengthen FP and adolescent SRH; im- prove pregnancy advisory centre services (34) Ipas, HRP Senegal 2010 Research/pilot programmes under way Addressing gender equality and women’s SRH needs Ipas, HRP Zambia 2008 Research/pilot programmes under way; scale-up initiated na Ipas, HRP South-East Asia Region Bangladeshb 2011 Not applicable Clinical norms/guidelines developed HRP European Region The Republic of Moldova (personal communica- tion, BR Johnson Jr, Scientist, WHO/HRP, 2012) 2005 Comprehensive abortion care (CAC) piloted in 3/12 perinatal centres (designated training sites); 1 district hospital; 1 youth-friendly clinic (medical abortion only); in- formation, education and counsel- ling (IEC) material disseminated nationally, including all 12 perinatal centres Implement national standards and guide- lines for CAC; revise training curricula based on new guidelines; develop indica- tors for national monitoring of the quality of abortion care; develop model outpatient services in selected sites Reproductive health training centre; James Tudor Founda- tion; HRP The Russian Federation 2009 Pilot testing Proposal to test an educational intervention on use of manual vacuum aspiration was delayed by HRP 2011 budget shortfall HRP Ukraine (36) 2008 Pilot testing in three sites; prepara- tion for scale-up Develop new protocol for comprehensive care for unwanted pregnancy (CCUP); clinical protocols; in-service training of mid- level professionals; three model CCUP clinics established; pre-/ post-abortion counselling introduced; develop and im- plement CCUP training curriculum HRP; Women’s Health and Family Planning; Swiss Agency for De- velopment and Cooperation; other NGOs na, not available. a Local research institution, Cellule de Recherche en Santé de la Reproduction en Guinée. b Full assessment was not done; clinical guidelines were identified as needed and developed in collaboration with HRP. Advocacy and technology to support the campaign A strategy toolkit has been developed to provide technical support for screening, case identifi- cation and contact tracing and includes protocols for monitoring and surveillance. Core annual elimination of congenital syphilis (ECS) data are submitted to WHO or the Joint United Nations Programme on HIV/AIDS (UNAIDS), through WHO regional offices. Programme impact is measured by the progress toward the target incidence of congenital syphilis (≤0.5 per 1000 live births). An ECS website provides tools and information at http://www.who.int/reproductivehealth/topics/rtis/cs_global_updates/en/index.html. HRP External Evaluation 2008−2012 Chapter 4 Page | 111 Progress Table 3 highlights progress in Latin America and the Caribbean (LAC), Africa and Asia. LAC countries were best prepared to embrace the initiative and adopted a plan of action in 2010, which has been integrated into national plans in 22 of 41 countries. Eleven countries reported target incidence rates for congenital syphilis of ≤0.5 per 1000 live births. In Africa, progress has been slow, owing to weak systems to deliver services and challenges, because of the culture of late attendance for antenatal care (ANC). In 2011, the initiative was launched in the Asia- Pacific region, with some success reported in India, Malaysia and Myanmar. Table 3 Stages 2 and 3: strategic approach, five African countries Ghana Guinea Malawi Senegal Zambia Law: relatively liberal Law: some restrictions Law: restricted Law: very restricted Law: restricted Stage 2: strategic development/pilot intervention Develop CAC services: • National standards, guidelines and pro- tocols (2004, 2006) • Update health in- formation systems to monitor and eval- uate abortion- related services • Train mid-level staff to provide abortion using MVA up to 12 weeks’ gestation Prevention of unwant- ed pregnancy/unsafe abortion included in Strategic plans for safe motherhood (2001– 2010); family planning (2010); health and development of ado- lescents/young people (2008–2012). Areas of focus include: • FP services • commodity security • sexual health educa- tion • improve PAC Develop national standards and guide- lines for provision of abortion MVA equipment added to Standard equipment list (2009); availability limited • Advocacy for legal reform initiated with creation of Coalition for Prevention of Un- safe Abortion (COPUA) • Media sensitization • Wide discussion of discordance between the Maputo Protocol (36) and the law Medical Association of Malawi to develop clini- cal protocols for CAC Goal: liberalization of current law to include grounds for rape/incest • Draft bill prepared by the Association of Women Lawyers of Senegal based on the Maputo Pro- tocol (36) • Advocacy task force formed to conduct awareness raising among parliamen- tarians, religious leaders, journalists and civil society Development of CAC services • National standards, guidelines and pro- tocols (2009) • CAC provided in 28 facilities in three of eight provinces • Operations re- search undertaken to train nurse prac- titioners and clini- cal officers to de- liver medical and surgical abortion services Stage 3: scaling-up • Partner with public, private and nongov- ernmental organiza- tions to expand ser- vices • CAC now available in 60 public and pri- vate facilities, in- cluding 12 hospitals • Incremental scaling- up to 18 more facili- ties; goal is national coverage • Given revisions to the law and lack of knowledge about changes, the team recommended: • develop clear legal guidance for applica- tion of the law • wide dissemination of information on le- gal indications for abortion • Ministry of Health has started to strengthen PAC and CAC as le- gally permitted • Infrastructure and equipment need to be assessed in 20 health centres in the capital, Lilongwe • MVA orientation workshops held • Plans are being developed to im- prove the quality of PAC, FP services and SRH education • In two study prov- inces, additional facilities have been added in partner- ship with Marie Stopes Interna- tional HRP External Evaluation 2008−2012 Chapter 4 Page | 112 Ghana Guinea Malawi Senegal Zambia Law: relatively liberal Law: some restrictions Law: restricted Law: very restricted Law: restricted Enabling environment • Medabon registered for use in pub- lic/private facilities • Misoprostol added to essential drug list for obstetric/ gynaecological use • Fixed price for abor- tion in public facili- ties • FP commodities in short supply • No legal reform ex- pected • Legal requirement that three physi- cians approve eve- ry abortion is a bar- rier • Medabon regis- tered but use lim- ited to obstetrics/ gynaecology; availability limited Status • Services have in- creased • Women still opt to self-induce with misoprostol (PAC is free; safe abortion costs US$35–40) • Lack of donor funds to support follow-up activities delayed implementing action plans • Delivery of CAC seems long way off • Efforts needed to improve access to FP services/commodities • Civil society repre- sentatives, support- ed by The Ministry of Health and Pre- vention and the Min- istry of Gender are advocating for legis- lative change re- garding abortion • Project being scaled up national- ly Effectiveness, efficiency and sustainability The willingness of development partners like the Global Fund to Fight AIDS, Tuberculosis and Malaria to support the integration of ECS into the HIV portfolio will help promote the initiative (see Table 4). Challenges include the persistence of vertical programmes (e.g. ANC, HIV), weak mid-level human resources skills, and inefficient logistical support. Technology transfer is needed to help scale up services, including strengthening of laboratory facilities, laboratory information systems, surveillance, and programme evaluation. Cultural and legal barriers, in- consistent funding and weak political will are threats to sustainability. HRP External Evaluation 2008−2012 Chapter 4 Page | 113 Table 4 Framework to enable access to comprehensive reproductive health and safe abortion care: the combined experiences of African and eastern European countries Areas for action Legal and cultural behaviour change Gendered atti- tudes and values Skills and competen- cies of health provid- ers Appropriate environ- ment to deliver safe abortion care Planning and regu- latory environment Objec- tives • Harmonize laws with international standards • Reduce stigma and discrimina- tion • Improve SRH literacy • Improve clinical and counselling skills and competencies • Ensure access to care • Provide adequate resources • Enable safe abortion care • Monitor access, safety, quality and outcome Targets • General public • Legislators • Political leaders • Religious leaders • Enable SRH education in schools for boys and girls • Develop SRH educators • Provide public education aimed at dis- pelling local myths and negative atti- tudes • Develop and imple- ment regulations so that middle-level pro- viders may deliver FP/safe abortion care • Develop teaching skills and training teams (trainer of trainers: in service; tertiary) • Modify curricula at basic, undergraduate and postgraduate levels, consistent with current knowledge and evidence • Enable continuing education for health- care providers • Add required drugs and equipment to es- sential drugs and equipment list • Upgrade facilities to provide care in a con- fidential, well- equipped environ- ment • Ensure geographic access • Ensure consistent supply of drugs, equipment and sup- plies • Maintain and replace equipment as neces- sary • Provide pre- and post-abortion coun- selling • Ensure affordable access to FP • Ensure compre- hensive needs assessment, which includes diverse input (strategic as- sessment stage 1) • Ensure adequate planning and identification of resources to en- able a successful pilot stage (stage 2) • Have a clear strategy for how the scale-up will be financed and implemented (stage 3) • Have a clear, integrated framework for monitoring safety, quality and out- come a Or creatively avoid a public challenge to the existing law, as was done in Bangladesh. Summary The ECS campaign promotes the integration STI and SRH services based on a country’s level of readiness. High disease burden but good infrastructure allowed LAC to embrace the double goal to eliminate both congenital syphilis and mother-to-child transmission of HIV, while other settings focused only on the ECS target. By shifting the focus to syphilis, a treatable condition, teams could be encouraged by their success, and reap the bonus of reduced HIV transmission. This campaign is an excellent example of the public good created by HRP, which brings new technology to bear on a persistent problem, with the tangible possibility of improving health and survival. Partnerships with academic institutions, nongovernmental organizations and other international agencies enable high-quality research to inform decision-making about what works, marrying skills in high-income countries to the needs of low- and middle-income countries, while building inter-country collaboration between low- and middle-income coun- tries to help each other. The achievement of target incidence rates for congenital syphilis of ≤0.5 per 1000 live births in 11 LAC countries is encouragement that the goal is not impossible. HRP External Evaluation 2008−2012 Chapter 4 Page | 114 Conclusions and recommendations Impact Tables 5 and 6 examine possible associations between these interventions and improvements in global health indicators. While achieving the first screen-and-treat ECS target is possible, as global access to ANC has increased, the third-trimester re-screen may not be feasible outside south east Asia and the LAC region. The higher observed contributions of women to reported HIV prevalence, despite stabilizing prevalence rates, may be a measurement artefact of im- proved antenatal surveillance, but we await the next round of indicators to decide whether new initiatives are needed to address the evolving situation. The stalled progress of abortion rates is cause for concern. Table 5 Potential for syphilis elimination in the 10 high-impact countries Country Estimated births per year (2010) (46) % of women mak- ing 1 and 4 antenatal visits (2006–2010) (46) % of ANC wom- en tested at first visit (2010) (47) Prevalence of syphi- lis in antenatal popu- lation (%) (47) Estimated still- births and (still- birth rate per 1000) (2009) (48) Bangladesh 3 038 000 53 and 23 na 0.6 (2008) 128 500 (36) India 27 165 000 75 and 51 65 0.3 (2010) 605 230 (22) Indonesia 4 372 000 93 and 82 na na 62 290 (15) Kenya 1 529 000 92 and 47 59 1.8 (2010) 34 130 (22) Madagascar 732 000 86 and 49 85 3.4 (2010) 14 590 (21) Mozambique 883 000 92 and 53 68 6.0 (2010) 25 560 (28) Rwanda 438 000 98 and 35 75 1.5 (2010) 9 620 (23) South Africa 1 059 000 97 and 87 75 2.2 (2010) 22 560 (20) United Repub- lic of Tanzania 1 862 000 88 and 43 54 4.2 (2008) 47 550 (26) Zambia 600 000 94 and 60 43 5.3 (2010) 14 380 (26) Total 38 625 000 52–98 and 23–87 43–85 0.6–6.0 (2008–2010) 964 410 (15–36) na, not available. HRP External Evaluation 2008−2012 Chapter 4 Page | 115 Table 6 Summary of ECS implementation efforts, by region Assessment Latin America and the Caribbean Africa Asia Rationale/ strategy The region adopted a plan of action in 2010; most countries have de- veloped strategies and plans for both ECS and MTCT of HIV Progress has been limited to the development of country strategic frameworks (e.g. Kenya, South Africa) The Asia Pacific United Nations Prevention of Parent-to-Child Transmission (PPTC) Task Force was launched in August 2011, to lead the initiative Development of screening tools Screening included in routine ANC; challenges reported in obtaining results in timely manner Screening demonstrated to be cost effective (range of cost per DALY saved: US$3.97–10.5) Pregnant women present late for ANC: >6 months’ gestation (e.g. Kenya, South Africa) Variations in antenatal syphilis screening practices; routine screening of all ANC clients done in few countries –Malaysia, Pa- pua New Guinea, Sri Lanka, Thailand (52) Development of guidelines Regional and national plans and clinical guidelines in place in 22 of 48 countries (e.g. Brazil, Cuba, Jamaica) (53) Lack of guidelines for service providers Lack of tracking systems for laboratory results – weak diag- nosis/follow-up systems (e.g. Kenya, South Africa) Monitoring and evaluation guide, fact sheets (12 countries) and costing tool developed (August 2011) Roll-out in field: efficiency Cuba has recorded the greatest success in the elimination initiative. Most countries reported 80% cov- erage on ANC and potential to provide the service; however, the quality of service varies across countries Limited training and supervision of field staff Unavailability and poor manage- ment of drugs and supplies (e.g. Kenya, South Africa) Overall weak implementation (e.g. Ghana) Limited data available on pro- cess issues High rates of first ANC attend- ance – 79% Monitoring effectiveness ECS indicators reported annually; surveillance and health information systems challenged by data quality (e.g. 16/48 countries not reporting indicators); legal and technical frameworks strengthened through WHO regional/country-initiated efforts Weak surveillance, monitoring and evaluation (e.g. Kenya, South Africa) WHO regional offices providing guidance on ECS indicators (e.g. Bangladesh, China, India, Thai- land) Reported reduction in syphilis serology among pregnant women in ANC in India and Myanmar Impact PMTCT coverage – 61% (2010) Syphilis testing coverage – 61% (57) 11 countriesa report a target for incidence of congenital syphilis of ≤0.5 per 1000 live births Antenatal syphilis prevalence: 0.1–0.4% in China, India, Malaysia; 6–7% in Papua New Guinea and Solomon Islands (52) Sustainability Declining political will and funding support Lack of sustained political com- mitment and advocacy (e.g. Kenya, South Africa) Need to scale up PMTCT pro- grammes to reverse the trend in infections in neonates DALY, disability-adjusted life-year. PMTCT, prevention of mother-to-child transmission. a Anguilla, Antigua and Barbuda, the Bahamas, Barbados, Canada, Cuba, Chile, Guadeloupe, Guyana, Panama and United States of America. Recommendations Input A clear policy is needed to guide research regarding male reproductive health, including the development of male contraceptives. Within this framework, it may be necessary to revisit how costs and benefits of a male contraceptive are measured, as successful avoidance of un- wanted pregnancy benefits not only the health of the female partner, but also the economic survival of the household and the community. HRP External Evaluation 2008−2012 Chapter 4 Page | 116 Process Strategies are needed to measure access to emergency contraception, and its utilization and impact. Integration of indicators into demographic and health surveys or other reproductive health surveys may be a place to start. Output/outcomes To support the safe abortion and ECS goals, monitoring of indicators should be included in the Millennium Development Goal and other universally accepted country reporting frameworks. Summary Although it is relatively small, the HRP team in Geneva is impressive in its capacity to identify and coordinate a large network of investigators, collaborators and experts, drawn from aca- demic and research institutions across all the WHO/United Nations regions of the world, capa- ble of addressing acute problems and developing long-term solutions to global SRH challenges. The cases demonstrate the global value of the Special Programme of Research, Development and Research Training in Human Reproduction, from problem identification (unsafe abortion) and problem clarification (the strategic approach), to generating new knowledge (male contra- ception) and marketing new products (emergency contraception), to piloting and roll-out of solutions to global problems (safe abortion; ECS), while monitoring the emergence of new knowledge (hormonal contraceptives and HIV) to maintain public trust in WHO/HRP as a relia- ble source of global SRH advice. HRP External Evaluation 2008−2012 Chapter 4 Page | 117 1 Introduction The World Health Organization’s (WHO) Department of Reproductive Health and Research (RHR) is the implementing body for the multi-agency (United Nations Development Fund (UNDP)/United Nations Population Fund (UNFPA)/United Nations Children’s Fund (UNICEF)/WHO/World Bank) Special Programme of Research, Development and Research Training in Human Reproduction (HRP). The year 2012 marks the 40th year of a continuous and growing contribution of the team’s expertise to the development and synthesis of new knowledge aimed at improving the quality and outcome of global human reproductive activi- ties. WHO is a respected source of reliable advice and guidance to countries and communities. RHR is best known for the development of normative guidance and technical support that in- tegrates the best available evidence to guide reproductive health policy, programme develop- ment and clinical practice. The WHO global strategy on reproductive health (1) provides the policy framework for RHR and thus the HRP agenda. The overarching aim is to accelerate progress towards attaining the highest achievable standard of sexual and reproductive health (SRH) for all, and this is embod- ied in Millennium Development Goals (MDGs) 4, 5 and 6 in particular. RHR organizes its work around five core aspects of SRH: 1. improving antenatal, perinatal, postpartum and newborn care; 2. providing high-quality services for family planning (FP), including infertility services; 3. eliminating unsafe abortion; 4. combating sexually transmitted infections (STIs), including HIV, reproductive tract infec- tions, cervical cancer and other gynaecological morbidities; 5. promoting sexual health. RHR recognizes the synergies of strengthening these five aspects of SRH and how activities in one area can impact the others. They also understand that, by strengthening existing health services, they help to ensure accessible entry points for new interventions (1). Technical assis- tance to countries employs action-oriented research and research-capacity strengthening, and the evidence generated is used to develop and improve norms and standards for care. The 2008–2012 HRP external evaluation aims to inform the decision-making process for the Special Programme and its constituents in the following areas: • the relevance and fulfilment of HRP’s objectives; • its efficiency and effectiveness; • its comparative advantage; • the impact and sustainability of its work. 2 Methods The case-study for the external evaluation was a desk review of peer-reviewed and WHO pub- lications. This included consultations with relevant programme staff and collaborators. The process was guided by the key norms and standards set out by the Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) Network on Development Evaluation (2). Where appropriate, the review considered and addressed: HRP External Evaluation 2008−2012 Chapter 4 Page | 118 • relevance, by determining how HRP activities address the priorities and policies of countries and whether the activities and outputs are consistent with the intended im- pacts and effects; • effectiveness, by exploring the extent to which stated objectives were achieved, mind- ful of barriers or facilitators to goal achievement; • impact, by summarizing the ecological associations of activities with global reproduc- tive health indicators. In addition: • as the work of HRP is intended to catalyse change and development in reproductive health care in target countries, the evaluation attempted to document what skill transfer has occurred to ensure the sustainability of intended benefits once initial in- terventions were concluded; • while economic efficiency was beyond the scope of these case-studies, where feasi- ble, the time frame for achieving the objectives was examined alongside possible bar- riers to timely implementation. Programme staff were consulted at the following stages: 1. understanding programme activities (April to June 2012) and guidance concerning possible country visits; 2. feedback on the draft report and exploration of qualitative feedback from end-users (Sep- tember to October 2012); 3. finalization of the report (November 2012). 2.1 Phase I: fact finding with WHO programme teams Initial meetings were held with key staff members in Geneva from 18 to 20 April and 19 to 22 June 2012, to develop an understanding of the programme activities and the principal areas of focus of the professional teams, which were (see Annex 1): 1. promoting FP; 2. preventing unsafe abortion; 3. controlling sexually transmitted and reproductive tract infections. 2.2 Phase II: desk review and consultation with stakeholders Based on phase I discussions, programme activities were selected by the consultant to demon- strate the leadership of the three teams in evidence generation, synthesis and programme im- plementation (Chapter 6 focuses on implementation research), to inform improvements in service delivery. During the evaluation, and in consultation with team leaders at WHO, stake- holders were consulted by e-mail for input or feedback on the draft report. 3 Rationale The 2003–2007 case-study focused on one of the core areas – improving antenatal, perinatal, postpartum and newborn care. It examined the synthesis of evidence regarding which antena- tal care (ANC) activities effectively improved pregnancy outcome. This was integrated into a revised ANC model, piloted and then field-tested in countries selected to represent culturally diverse regional and service-delivery capabilities. HRP External Evaluation 2008−2012 Chapter 4 Page | 119 A field visit to Thailand enabled observation of the national roll-out of this model and provided an opportunity to demonstrate how effective application of reproductive health research in- fluences policy and, ultimately, practice. The case-study for the 2008–2012 review examined three other areas of the Programme’s mandate – to improve FP; prevent unsafe abortion; and prevent and control sexually transmit- ted and reproductive tract infections. The review aimed to provide a synopsis of HRP’s work in action, across a range of settings and encompassing the processes of evidence generation, with synthesis of the evidence into programme guidance, through to national and internation- al roll-out of proven strategies aimed at improving SRH. To this end, activities in multiple coun- tries, not just one, were used to demonstrate the uptake of HRP guidance and leadership in SRH. 4 Findings 4.1 General The WHO strategic approach to strengthening sexual and reproductive health policies and pro- grammes is a strategic planning, policy and progamme implementation tool that HRP teams use to evaluate and improve reproductive health services (3). It consists of three stages: 1. a strategic assessment to identify and prioritize needs; 2. introduction of interventions on a small scale, to address priority needs identified from the first stage, aimed at demonstrating their acceptability, feasibility and effectiveness; 3. scaling up proven interventions to expand their benefits to more people and strengthen institutional capacities. An essential feature of the tool is its multidisciplinary, participatory framework, which enables a wide range of stakeholders, including beneficiaries, to participate in the evaluation and deci- sion-making process. Other characteristics include: • a staged implementation process, which links assessment, pilot-testing and scaling up; • a systems framework that highlights relevant factors for decision-making about ser- vices; • a reproductive health philosophy of reproductive rights, gender equity and empow- erment; • a focus on improving equitable access to and quality of care, so that services are client centred and responsive to community needs; • a participatory process, which consider the concerns of all relevant stakeholders; • country ownership of the process and the results (4). The process includes five overarching activities, derived from the WHO global Reproductive health strategy (1): (i) strengthening health-systems capacity; (ii) improving information for priority setting; (iii) mobilizing political will; (iv) creating supportive legislative and regulatory frameworks; and (v) strengthening monitoring, evaluation and accountability. Elements of the strategic approach will be demonstrated by the case-studies. The FP case- study will summarize some of the challenges of generating new evidence (male contraception) and getting successes to market (emergency contraception), while maintaining a strong moni- toring role that efficiently responds to new evidence (HIV acquisition and hormonal contracep- tion). The safe abortion case-study demonstrates application of the strategic approach to a sensitive problem, enabling incremental, but certain progress toward improving health and HRP External Evaluation 2008−2012 Chapter 4 Page | 120 safety. The initiative to eliminate mother-to-child transmission (MTCT) of syphilis reveals how evidence generation, health systems research, surveillance, monitoring and evaluation com- bine to tackle a long-standing public health problem. 4.2 Improving family planning: generating new evidence, marketing new products and protecting health and safety 4.2.1 Introduction The FP agenda aims to improve the quality of FP methods and services. By utilizing epidemio- logical, social, behavioural and operations research, teams document unmet need; identify sociocultural barriers and health systems constraints that limit access to services, and test ap- proaches to mitigate them; develop and assess the role of mid-level providers in improving access to services; and develop and test new tools to monitor how well gender issues and hu- man rights are respected in FP services. RHR is best known for the development of evidence-based FP guidelines and tools for service providers. The Medical eligibility for contraceptive use (MEC) now in its fourth edition (2009) (5) and translated into over 15 languages and covering 17 FP methods, was awarded a first prize at the British Medical Association (BMA) Book Awards2 in 2011, in the Obstetrics and Gy- naecology category (6). To ensure that guidelines are based on current knowledge, HRP monitors emerging evidence. This process has resulted in two consultations since the 2009 release of the MEC guidelines (5), concerning the use of hormonal contraceptives and related risk of venous thromboembolism (VTE) (7) and HIV, respectively. Revised guidelines were issued concerning VTE (see Annex 2), while the HIV risk did not generate new guidelines and will be detailed below. The marketing of levonorgestrel for emergency contraception (EC), and the efforts to develop a male contra- ceptive, will also be discussed. 4.2.2 Creating new knowledge: developing a male contraceptive Background In recognition that inadequate attention has been paid to the SRH needs of men, in 2010 RHR established a working group on Men and Sexual and Reproductive Health, to formalize strate- gic programming in relation to men’s SRH needs (8, 9). Areas of work included understanding men’s roles in sexual decision-making and behaviour and their shared burden and responsibil- ity for pregnancy prevention, including contraception. As part of improving the science of male reproductive function (andrology), RHR has been coordinating development of a hormonal regimen for male contraception (8). How male hormonal contraception works Hormonal methods of regulating male fertility utilize exogenous androgen (e.g. testosterone) to reduce the body’s ability to produce sperm (spermatogenesis). Suppression of sperm pro- duction is achieved by inhibiting hormones secreted by the pituitary gland (follicle-stimulating hormone (FSH) and luteinizing hormone (LH)), which in turn inhibit the production of testos- terone by cells in the male testes. High levels of androgens in the testis are required for normal spermatogenesis; thus, very low levels of androgens in the testis result in significant suppres- 2 The awards encourage and reward excellence in medical publishing; there are prizes in 21 categories, including an overall BMA Medical Book of the Year from among the category winners. HRP External Evaluation 2008−2012 Chapter 4 Page | 121 sion of, or no detectable, sperm. Suppression is achieved in most men when testosterone is administered with a progestin (10). A combined long-acting injectable androgen, testosterone undecanoate (TU), with a long- acting progestin, norethisterone enantate (NET-EN), was tested for sperm suppression and demonstrated suppression of spermatogenesis in nearly all men (11). A dose of 1000 mg TU plus 200 mg NET-EN administered every 8 weeks was considered to be preferable to TU alone or to 750 mg TU plus NET-EN for contraception regimens, owing to more complete suppres- sion of gonadotropins and spermatogenesis; this formed the basis for a multicentre trial (12). No major adverse events that would preclude development of the combination as a reversible male contraceptive were reported. Study objectives and methods The study was designed to evaluate whether the combination of NET-EN and TU would be a safe, effective, reversible and acceptable male contraceptive (13). The two drugs were admin- istered by injection every 8 weeks. The study consisted of a suppression phase (up to 26 weeks), an efficacy phase (up to 56 weeks) and a recovery phase. In the initial drug- exposure phase, which has now been discontinued, men were monitored for suppression of spermatogenesis. Men who demonstrated adequate suppression proceeded into a 12-month contraceptive efficacy phase, which has also now been discontinued. This was followed by a recovery phase, which is still ongoing, in which the men are monitored for return to sperm levels that are indicative of fertility. Men whose sperm concentrations were not adequately suppressed initially, or whose sperm levels rose above the threshold during the efficacy phase, or who wished to withdraw from the study for any reason, also entered the recovery phase. Trial monitoring, expected side-effects, interim findings and the decision to stop the trial The trial was subject to annual reviews. At WHO, these were conducted by the research pro- ject review panel (RP2) and the WHO Ethical Review Committee (ERC). There was also periodic review by an independent data safety and monitoring committee (DSMC) and individual site ethical committees. These committees are intended to ensure that the rights, safety and well- being of study participants are protected. The WHO RP2 met in 2011 to review the status of the study and the available combined safety information from all sites. Expected side-effects of androgen and progestin administration included acne, increased libi- do, injection-site pain, weight gain and mood changes. Participants were informed of the pos- sibility of side-effects at enrolment but some occurred more often than expected. Reported side-effects of concern were depression and other mood changes, increase in sexual desire, and injection-site pain. Mood changes and increased libido were reported at a higher frequen- cy than anticipated. Eight “serious adverse events” occurred, two of which were judged to be either possibly or probably related to the study regimen; the other six have been classified as unrelated to the study regimen. The reviewers determined that the potential for side-effects was greater than the benefit of the study drug to the male participants (14). Given the reported side-effects, the WHO panel questioned whether this drug combination can be successfully developed and marketed as a male contraceptive, and opted to stop the trial. The RP2 recommended that injections of the two study drugs (TU and NET-EN) should not continue and all active participants should transi- tion to the recovery phase. The ERC accepted this recommendation. HRP External Evaluation 2008−2012 Chapter 4 Page | 122 How many men are currently in the study and was the treatment effec- tive? A total of 321 men were enrolled at 10 research sites in eight countries.3 At 31 March 2011, 110 men had completed the full 12-month effectiveness phase. When the study was stopped, 103 men were receiving injections in the suppression or efficacy phase and were transferred to the recovery phase. All men in the initial suppression or efficacy phases attend two-monthly clinic visits until recovery of sperm to levels considered as fertile is demonstrated. By June 2012 (8 weeks post injection plus a scheduled 52-week recovery period), only five men had not recovered fully. Data collection should conclude by November 2012, when all sites have had their “close-out” visits. Sera from all sites will then be sent to a central laboratory for standard- ized analysis of hormones. Integration of the laboratory data with the clinical data and report- ing is expected in 2013 (personal communication, D Colvard, Deputy Director for Programs, CONRAD, 17 September 2012: WHO-CONRAD Male contraceptive trial: update). An earlier interim analysis indicated that the two hormones suppressed sperm production in nearly all men. Very few pregnancies occurred during the 12-month efficacy phase, when par- ticipants were asked to rely only on the study regimen for contraception. Anecdotally, many men were satisfied with the method, as were their partners. Male participants and female partners completed acceptability questionnaires several times during the study, and these da- ta will be analysed when the study ends. Conclusions: the future of male hormonal contraception development Developing a new product is often a long and challenging process. CONRAD4 and WHO have supported research in male contraception, and while this drug combination in the current dos- age will not move forward as a contraceptive method, this clinical trial provides valuable in- formation on its effectiveness, safety and acceptability. Research is expected to continue on the development of a male contraceptive, whether hormonal or non-hormonal. When unexpected findings arise in routinely monitored clinical trials, the first priority is to pro- tect the health and well-being of participants. The WHO recommendation applies to this TU/NET-EN combination, not to the drugs used alone for their approved indications or in other combinations. The side-effects of concern observed during this study are not seen to the same extent when the drugs are used in their approved indications and dosing regimens. The concerns about the risk–benefit ratio of this particular regimen should not apply to the development of other hormonal regimens for men. While all medications carry some risk, re- searchers and clinicians must ensure that the risks do not outweigh the benefits, especially in research. For female contraception, the risks of birth control are compared to the risks of pregnancy and its complications. The benefit of preventing pregnancy and its possible prob- lems outweighs the risk of taking hormonal contraception for most women. With male contra- ceptive methods, the man is asked to bear the method’s risks without the direct benefit of preventing the risk of pregnancy. As a result, the risks of male hormone contraceptive meth- ods may seem high compared to the benefits. However, a male contraceptive addresses the need of couples where women’s health concerns contraindicate the use of hormonal or other methods, and, secondly, a male option other than condoms addresses important issues about shared responsibilities in FP. There may be a need to revisit the risk–benefit assessment asso- 3 The sites were: Melbourne and Sydney, Australia; Santiago, Chile; Halle and Munster, Germany; New Delhi, India; Jakarta, Indonesia; Bologna, Italy; Edinburgh and Manchester, United Kingdom of Great Britain and Northern Ire- land. 4 CONRAD is a reproductive health research organization within the Department of Obstetrics and Gynecology of Eastern Virginia Medical School, United States of America. HRP External Evaluation 2008−2012 Chapter 4 Page | 123 ciated with shared responsibility, as, while it is the woman who physically carries the child, it is the family unit not just the woman, that benefits from the successful avoidance of an unin- tended pregnancy. 4.2.3 Getting a new product to market: emergency contraception Background EC regimens have been available since the early 1990s (15), for use within 120 hours of unpro- tected sexual intercourse, to prevent pregnancy before it occurs, especially in circumstances such as rape, incest or non-use or failure of other methods (e.g. burst condom). EC acts pri- marily by preventing ovulation or by interfering with sperm so that they cannot fertilize the ovum. Because EC is not an abortifacient and cannot terminate an established pregnancy, it is acceptable in countries with highly restricted abortion laws, such as in Latin America (e.g. Ar- gentina, the Bolivarian Republic of Venezuela, Brazil, Colombia, El Salvador) and former British colonies where the 1860s Offences Against the Person Act remain in force (e.g. Jamaica, Kenya, Pakistan) (16). HRP has played a pioneering role in the development of EC and in making it easier and safer to use, and more widely available to women worldwide. HRP has conducted large multicentre randomized trials demonstrating the safety, efficacy and dosing of levonorgestrel for EC. Sub- sequent work from HRP has contributed to supporting the successful registration and in- creased utilization of single-dose levonorgestrel as an emergency contraceptive in several countries. EC is commonly packaged as two levonorgestrel pills, although women are instructed to take both pills at once (17), owing to the dosing studies conducted by HRP. WHO/HRP has collaborated with international partners such as the International Federation of Gynaecology and Obstetrics, International Planned Parenthood Federation and International Consortium for Emergency Contraception, to educate the public about the safety and use of EC and improve access to EC through advocacy to enable registration of EC products, preferably as a non-prescription over-the- counter (OTC) item, ensuring that EC is available to women at greatest risk of unwanted pregnancy (18). As part of their international commitments to re- duce unwanted pregnancy and prevent pregnancy-related deaths and illnesses, governments worldwide are encouraged to take the essential first steps to ensure access to EC for every woman. Table 1 summarizes the status of EC availability in 2010. Of 189 countries worldwide, EC is available without prescription in 103 (54%) and by prescription in 23 (12%). HRP External Evaluation 2008−2012 Chapter 4 Page | 124 Table 1 Countries with access to emergency contraception, by availability and region Region Available without prescription (pharmacist/OTC) Prescription required North America Canada, Mexico United States of America (nine states no restriction)a Caribbean Antigua and Barbuda, Aruba, the Bahamas, Barbados, Belize, Cura- cao, the Dominican Republic, French Guiana, Haiti, Jamaica, Puerto Rico, Saint Lucia, Trinidad and Tobago Latin America Argentina, the Bolivarian Republic of Venezuela, Ecuador, El Salva- dor, Guatemala, Nicaragua, the Plurinational State of Bolivia, Brazil, Chile, Colombia, Cuba, Paraguay, Peru Western Europe Austria, Belgium, Denmark, Iceland, Finland,b France, Greece, Lux- embourg, the Netherlands, Norway, Portugal, Spain, Sweden, Swit- zerland, the United Kingdom of Great Britain and Northern Ireland Germany, Italy, Lithuania Eastern Europe Albania, Armenia, Azerbaijan, Belarus, Estonia, Kazakhstan, Kyrgy- stan, Latvia, Lithuania, Poland, Romania, Serbia, Slovakia, Slovenia, Tajikistan, Ukraine Bulgaria, Czech Republic, Georgia, Hungary, the Russian Federation Africa Algeria, Benin, Burkina Faso, Cameroon, the Congo, Côte d’Ivoire, the Democratic Republic of the Congo, Egypt, Ethiopia, Gabon, Ghana, Guinea, Kenya, Lesotho, Libya, Madagascar, Mali, Mauritius, Moroc- co, Namibia, the Niger, Nigeria, Senegal, South Africa, Togo, Tunisia, Uganda, the United Republic of Tanzania, Zambia Botswana Middle East Cyprus, the Islamic Republic of Iran, Israel, Mauritius, Saudi Arabia, Turkey, Yemen Lebanon Asia China, Hong Kong, India, Japan, the Lao People’s Democratic Repub- lic, South Korea, Sri Lanka, Thailand, Viet Nam Bangladesh, Indonesia, Malaysia, Myanmar, Pakistan, Singapore, Taiwan Oceania Australia, French Polynesia, New Zealand Total 103 23 a Some states restrict use for adolescents aged <18 years or <16 years. b Not for adolescents aged <15 years. International collaborative efforts to promote and expand access to emer- gency contraception Social marketing (5, 20) has been promoted for distributing EC, given its success with other reproductive health products such as condoms and oral contraceptives. The process operates through existing commercial channels, to place the products on the shelf along with other health- and household-related items. The International Consortium for Emergency Contracep- tion (ICEC) has collaborated with HRP to monitor the social marketing of EC. In some settings, the products may be subsidized to ensure that the price is affordable for the intended users. Strategies include: • use of a commercial partner to distribute the product through commercial outlets, while a nongovernmental organization (NGO) provides health services or support such as advocacy with policy-makers, training of providers and outreach to women; • the NGO registers, imports and distributes the product, either through commercial outlets or through branded or “franchised” clinics, pharmacies or drug sellers (21). An ICEC review of social marketing in a range of countries, including the Bolivarian Republic of Venezuela, Egypt, Ethiopia, India, Indonesia, Pakistan and Viet Nam, found that successful pro- grammes, based on demonstrated high-volume sales, had some common characteristics, in- cluding: • government support for advertising and public education (e.g. hotlines, dealer incen- tives, promotion with pharmacists and doctors, catchy radio advertisement cam- paigns); HRP External Evaluation 2008−2012 Chapter 4 Page | 125 • government approval of OTC sales; • government support of social marketing as a population policy strategy; • government distribution of EC within its FP programme; • access to large urban populations; • introduction of a single-pill product; • affordable price and convenience (especially when available at local pharmacies); • acceptability by women, and thus promotion by word of mouth; • low use of other contraceptive methods, especially long-acting methods; • high rates of unprotected sexual intercourse; • convenience and discretion of private sector marketing; • consumer education campaigns and promotional activities; • promotion among providers in social franchise clinic networks; • improved awareness by shop owners, staff, doctors, pharmacists and customers. Barriers to inclusion or success of EC in social marketing and other commercial distribution programmes are: • delays in the authorization process to have EC registered; • lack of public sector funding for the introduction and promotion of EC; • limited programme scope, because of political/legislative barriers; • church protests, which include branding the product as an abortifacient; • court decisions declaring EC acts as an abortifacient and prohibiting public sector dis- tribution; • requirement for a prescription; • lack of donor funding to socially market EC in many countries; • lack of client knowledge that EC is an option; • stock-outs. As EC has become more widely available and accepted, e.g. in Mexico and India, direct uptake by the commercial sector has reduced the need for social marketing agencies. Emergency contraception: access and impact EC is an acceptable post-coital method to prevent unwanted pregnancy, which can help reduce the number of unsafe abortions, especially in settings where abortion is not legal. Interven- tions designed to increase access to EC have not shown significant reductions in population- level rates of unintended pregnancy (22) or abortion, nor has its availability been associated with increasing promiscuous behaviour or unprotected sexual intercourse (23). High rates of unprotected sexual intercourse and rare use of EC probably limit population-level impact (24). If social marketing enables not only EC use but behaviour change toward more appropriate use of other contraceptive methods, attributing EC’s contribution to preventing unwanted preg- nancy may be difficult. Routine measurement of knowledge, access and utilization of EC should be integrated into reproductive health or demographic and health surveys, to document its market penetration. While EC is a potentially valuable reproductive health product, like access HRP External Evaluation 2008−2012 Chapter 4 Page | 126 to safe abortion, there are political hurdles to overcome (25). Social marketing has the poten- tial to expand access to and use of EC, especially when complemented with policy advocacy and educational activities directed at all stakeholders. 4.2.4 Monitoring product safety: hormonal contraception and risk of HIV acquisition Background Hormonal contraceptives include oral contraceptives pills (OCPs), injectables, patches, rings and implants. They are highly effective in preventing pregnancy, contribute to global public health targets such as reducing maternal morbidity and mortality, and translate into wider so- cial and economic benefit for the family and community. The 2009 revision of the MEC (5) re- ports that hormonal contraceptives are safe for use by women who are at high risk of, or living with, HIV. RHR monitors new research evidence to keep guidelines consistent with current knowledge. The emergence of a growing body of evidence suggesting an association between hormonal contraceptive use and risk of HIV infection acquisition among HIV-negative women, disease progression among women living with HIV, and transmission of HIV from women to non- infected male partners (26) suggested the need to review the 2009 guidance. Methods In collaboration with external experts and partners, HRP compiled three systematic reviews from peer-reviewed papers published up to 15 December 2011, focusing on findings published since the MEC revision meeting in 2008. Employing the grading of recommendations, assess- ment, development and evaluation (GRADE) approach, they examined the evidence regarding: • hormonal contraception and acquisition in women who are HIV negative; • hormonal contraception and disease progression in women who are HIV positive; • hormonal contraception and transmission from women who are HIV positive to men who are HIV negative. GRADE evidence profile summaries (see Annex 3) were peer reviewed by an advisory commit- tee and circulated to a team of experts ahead of a technical consultation of 75 participants from 18 countries, representing 18 agencies, who met from 31 January to 1 February 2012, to consider the findings. The multidisciplinary panel included clinicians, epidemiologists, re- searchers, programme managers, policy-makers, guidelines methodologists, reproductive biol- ogists and pharmacologists. Stakeholders included experts in international FP and HIV, as well as women’s health advocates. After careful review of the evidence and extensive discussion, recommendations were reached through consensus. The process was reviewed by the WHO Guidelines Review Committee5 on 15 February 2012. The recommendations were approved and released the following day and are posted on the WHO website for easy availability to the global public (27, 28). Summary of the evidence Biological studies 5 The Guidelines Review Committee is responsible for ensuring that all WHO recommendations are based on the best available evidence and have been developed in a transparent, unbiased and clearly reported manner. HRP External Evaluation 2008−2012 Chapter 4 Page | 127 Several biological mechanisms by which individual methods of hormonal contraception could influence HIV acquisition, progression and transmission were considered. Reviewers agreed that it remains unclear which, if any, are clinically relevant, as findings were inconsistent within and across contraceptive types. Different hormonal contraception could alter these factors in different ways and this depended on whether the preparations contained either or both estro- gen or/and progestogen. Potential mechanisms included alteration of systemic and local im- mune responses and changes in the genital tract environment. Epidemiological studies The epidemiological studies mostly assessed combined oral contraceptives (COCs) or proges- togen-only injectable contraceptives, including depot medroxyprogesterone acetate (DMPA) and NET-EN, and their relationship to acquisition of HIV infection among women who are HIV negative and at high risk of HIV, disease progression in women who are HIV positive or trans- mission from women who are HIV positive to men who are HIV negative, with the largest body of evidence related to disease progression. Acquisition in women who are HIV negative Twenty prospective studies assessed the risk of HIV acquisition among women who are HIV- negative. No significant association was found between OCP use and HIV infection, although point estimates varied and several had limited statistical power. Evidence on injectable contra- ception was mixed. While no statistically significant association was found with NET-EN, stud- ies reported mixed results, varying from a significant increase in risk (48–100%) to no associa- tion for DMPA. Disease progression in women who are HIV positive Of 10 observational studies of various hormonal contraceptives and HIV disease progression (measured by mortality, time to CD4+ cell count <200 cells/mm3, initiation of antiretroviral therapy (ART), increased HIV-RNA (ribonucleic acid) viral load, or decreased CD4+ cell count), only one found a statistically significant association in a comparison of individuals using hor- monal contraception with those using a copper intrauterine device (IUD). This study had methodological difficulties, including method switching and loss to follow-up, and the GRADE rating for this body of evidence was “low”. Transmission from women who are HIV positive to men who are HIV negative One study suggested a 2–3-fold increase in risk of female-to-male HIV transmission among us- ers of injectable but not oral hormonal contraceptives. While the study had methodological strengths (statistical adjustment for multiple potential confounders, low loss to follow-up, fre- quent follow-up visits, large population studied), it had limited statistical power, owing to the small number of new HIV infections in men. The GRADE rating for the body of evidence on fe- male-to-male HIV transmission was “low” for injectable contraception and “very low” for oral contraceptives. Recommendation As most concern focused around progestogen-only injectable contraception and risk of HIV acquisition in women, the experts decided that the available evidence did not establish a clear causal association between injectable contraception and HIV acquisition, but did not definitive- ly rule out a possible effect. They agreed that use of hormonal contraceptives should remain unrestricted but that further research was needed and emerging evidence needed to be close- ly monitored. HRP External Evaluation 2008−2012 Chapter 4 Page | 128 Women at high risk of HIV infection were advised to: • continue to use all existing hormonal contraceptive methods without restriction; • access and use condoms, male or female, and, where appropriate, other measures to prevent and reduce their risk of HIV infection and STIs. Women living with HIV infection were advised to: • continue to use all hormonal contraceptive methods without restriction; • consistently and correctly use condoms, male or female, to prevent HIV transmission to non-infected sexual partners. Rodriguez et al modelled the risk of HIV acquisition against the risk of maternal death in four African countries (Chad, Kenya, South Africa and Uganda) if women were to stop using proges- togen injectable hormonal contraception out of fear of acquiring HIV infection (29). The op- tions included no method, an IUD or COCs. The authors concluded that if less than 70–100% of current users do not switch to an IUD or COC, an additional nine maternal deaths will occur for every case of HIV that is averted. This risk must therefore be considered in the counselling of women in settings where both HIV and maternal mortality risks are high. HRP will continue to monitor the evolving evidence and the team can be relied on to adjust their recommendations in light of the evidence, as necessary. There is, however, an indication of an emerging need to consider and develop alternatives for these populations. 4.2.5 Safe abortion services: application of the WHO strategic approach to unsafe abor- tion – experience in sub-Saharan Africa, eastern Europe and Asia Background While the global abortion rate declined from 35 per 1000 women aged 15–44 years in 1995 to 29 per 1000 in 2003, it seems to have stalled at 28 per 1000 in 2008 (30). In low- and middle- income countries in 2008, most abortions remained unsafe (55%), reaching 97% in Africa where access to safe abortion is generally restricted, with over 2000 abortion-related deaths every year (31). In eastern Europe, where abortion is legal, limited access to contraceptives results in overuse of abortion, and unacceptable morbidity and mortality. In Bangladesh, menstrual regulation (MR),6 as distinct from abortion,7 was included in the na- tional FP programme in 1979 to address contraceptive failure. MR procedures may be per- formed up to 10 weeks from the last menstrual period (LMP). When MR was introduced, no pregnancy test was done and uterine contents were not examined following MR. This legal grey area concerning pregnancy status was used by Bangladesh to make MR widely and legally available. In 2011, Bangladesh requested HRP technical assistance to revise their national menstrual regulation guidance in line with WHO safe abortion guidance (33). The WHO strategic approach is a useful tool for tackling sensitive problems in a dispassionate way and has been applied to the prevention of unsafe abortion with good effect. Of 15 coun- tries that have used the strategic approach to address unsafe abortion,8 11 were active from 2008 to 2011 and form the basis for this case-study. These countries represent a range of legal settings, ranging from unfavourable to liberal, and demonstrate how challenges can be articu- 6 Defined as “a procedure to make the menstrual cycle regular if menstruation is absent for a short duration” (33). 7 Defined as “interruption or termination of pregnancy after implantation of the blastocyst in the endometrium and before the fetus has attained viability” (32). 8 Bangladesh, the former Yugoslav Republic of Macedonia, Ghana, Guinea, Kyrgyzstan, Malawi, Mongolia, the Re- public of Moldova, Romania, the Russian Federation, Senegal, Sierra Leone, Ukraine, Viet Nam and Zambia. HRP External Evaluation 2008−2012 Chapter 4 Page | 129 lated and solutions developed to improve access to safe abortion services regardless of the environment. The status of country experiences is summarized in Table 2. Table 2 Countries applying the WHO strategic approach to safe abortion, by stage of the process, 2012 WHO Region Country Initial year Strategic stage in 2011/2012 Objective/output/outcome Partners African Region Ghana 2005 Research/pilot programmes under way; scale-up initiated Develop national standards and guidelines Ipas; HRP Guinea 2009 Dissemination workshop 2010; no further progress Improve access to legal abortion; inform a national effort to reposition FP HRP; CERRU- GUIa Malawi 2009 Research/pilot programmes under way (provision of DMPA by com- munity health workers) Revise sexuality education curricula; strengthen FP and adolescent SRH; im- prove pregnancy advisory centre services (34) Ipas, HRP Senegal 2010 Research/pilot programmes under way Addressing gender equality and women’s SRH needs Ipas, HRP Zambia 2008 Research/pilot programmes under way; scale-up initiated Development of national standards and guidelines Ipas, HRP South-East Asia Region Bangladeshb 2011 Not applicable Clinical norms/guidelines developed HRP European Region The Republic of Moldova (personal communica- tion, BR Johnson Jr, Scientist, WHO/HRP, 2012) 2005 Comprehensive abortion care (CAC) piloted in 2/12 perinatal centres (designated training sites); 1 district hospital; 1 youth-friendly clinic (medical abortion only); in- formation, education and counsel- ling (IEC) material disseminated nationally, including all 12 perinatal centres Implement national standards and guide- lines for CAC; revise training curricula based on new guidelines; develop indica- tors for national monitoring of the quality of abortion care; develop model outpatient services in selected sites Reproductive health training centre; James Tudor Founda- tion; HRP The Russian Federation 2009 Pilot testing Proposal to test an educational intervention on use of manual vacuum aspiration was delayed by HRP 2011 budget shortfall HRP Ukraine (36) 2008 Pilot testing in three sites; prepara- tion for scale-up Develop new protocol for comprehensive care for unwanted pregnancy (CCUP); clinical protocols; in-service training of mid- level professionals; three model CCUP clinics established; pre-/ post-abortion counselling introduced; develop and im- plement CCUP training curriculum HRP; Women’s Health and Family Planning; Swiss Agency for De- velopment and Cooperation; other NGOs na, not available. a Local research institution, Cellule de Recherche en Santé de la Reproduction en Guinée. b Full assessment was not done; clinical guidelines were identified as needed and developed in collaboration with HRP. The WHO strategic approach and safe abortion guidance were introduced to anglophone (2007) and francophone African (2008) stakeholders at two workshops. Following this, five countries (Ghana, Guinea, Malawi, Senegal and Zambia) requested support from WHO and/or Ipas for national strategic assessments on unintended pregnancy and abortion. Countries had to firstly submit a proposal to HRP outlining their needs. After approval by WHO’s Research Ethics Review Committee, the process began. Country experiences Since 2005, Ipas has partnered with HRP to support interventions to reduce unsafe abortion in Africa. HRP facilitated six strategic assessments from 2008 to 2011 and assisted Bangladesh to develop clinical guidelines (2011). In Ghana and the Republic of Moldova, follow-up activities HRP External Evaluation 2008−2012 Chapter 4 Page | 130 to the 2005 strategic assessment were supported by HRP. As data were being collected and/or analysed in Kyrgyzstan and Sierra Leone during the review, these experiences will not be in- cluded. Stage 1: national strategic assessments Country assessment teams (representatives of ministries of health; national and international NGOs; health-care providers; youth and women’s advocates; and religious leaders, among others) conducted qualitative, in-depth individual and group interviews (personal communica- tion, E Jackson, Medical Officer, WHO/HRP, 2012). Male and female respondents were select- ed from among policy-makers, programme managers, women seeking reproductive health services and community members, and asked about unintended pregnancy and abortion; ac- cess to and availability of safe abortion services; quality of SRH services; policy; institutional capacity; laws; regulations; and human rights and health resources (human, physical, financ- ing). Findings and recommendations were presented to stakeholders at national dissemination workshops, who reviewed, refined and prioritized the recommendations. Development of ac- tions plans followed. Major findings from assessments in the five African countries and the Republic of Moldova Despite major cultural, political and economic differences, African countries identified similar barriers to safe abortion and pregnancy prevention, including lack of standards and guidelines for CAC, inadequate training of abortion providers and other reproductive health staff, poor monitoring of services and inadequate quality of care. Other common findings but important differences are detailed below. Legal restrictions on abortion/access to contraceptives While a favourable legal environment exists in Europe regarding access to abortion services, limited access and negative attitudes to contraception in Eastern Europe meant that abortion was routinely used to limit unintended pregnancies, with outdated practices leading to unsafe abortion, morbidity and mortality. Abortions laws had been liberalized in Ghana and Zambia; however, stipulations that services could only be provided by physicians, or that three doctors had to approve every abortion re- quest in writing, effectively limit access and prevent expansion of services by training mid-level providers. In countries that legally permitted abortion for specific indications, these grounds were rarely known, leading to an assumption that all abortion was illegal. While a 2000 revi- sion to the law in Guinea allowed abortion in cases of rape, incest or fetal anomaly, or to pre- serve a woman’s life, none of the community members, health-care providers or policy-makers interviewed knew these legal indications for abortion. A similar lack of knowledge was seen in Ghana and Zambia, where laws had been liberalized much earlier. Guinea, Malawi and Senegal had highly restricted access to abortion, limiting access to safe services, particularly for the poor, with demand for post-abortion care (PAC) exceeding supply. In Guinea, hospital committees must approve abortion requests but only two hospitals had such committees, both in the capital, Conakry. Assessors from Malawi and Senegal noted the lack of harmony between laws that only permit abortion to save the life of the woman, and the Maputo Protocol (2003) (36), ratified by Senegal (2004) and Malawi (2005), which calls for provision of abortion on broad-based legal grounds. Systems of clandestine abortion provision had high associated costs for women Even where safe legal services exist, women resort to clandestine care, often from doctors at privately negotiated prices, sometimes under the guise of PAC. The price depends on gestation HRP External Evaluation 2008−2012 Chapter 4 Page | 131 and provider qualifications. While generally safer than abortion provided by traditional practi- tioners, the higher cost and limitation to urban areas restrict access for poor, rural, young and otherwise vulnerable women. In Ghana and Zambia, some women buy drugs such as miso- prostol from pharmacists or “chemical sellers”. Inadequate post-abortion care and weaknesses in family planning and sexuality education Although PAC was variably available, policy, practice and quality of care issues such as lack of, or poorly maintained, manual vacuum aspiration (MVA) equipment, or limited availability of services resulting in delays in uterine evacuation, often result in practitioners resorting to higher-risk sharp curettage procedures, which are only available in hospitals, further limiting access. PAC services often lacked contraceptive counselling. Low contraceptive use was the main determinant of unsafe abortion. Barriers include limited infrastructure to provide contraception; stock-outs or lack of contraceptive commodities; myths and misperceptions about contraceptive use; gender inequity; and adolescent sexuality. Countries with severe legal restrictions on abortion are limited to strengthening FP and post- abortion contraception. Despite sexuality education in some schools, many teachers felt ill prepared to teach the subject, especially where open discussion of sexual intercourse and sex- uality was largely taboo. In Bangladesh, MR is performed by MVA by government-trained doctors and specially trained paramedics (family welfare visitors and female subassistant community medical officers). While several MR guidelines have been in use, the Ministry of Health saw the need for com- prehensive National menstrual regulation services guidelines (32) to standardize norms, policy, regulations and performance descriptors using evidence-based clinical decision-making strate- gies. Religious, social and cultural contexts Abortion has been highly stigmatized and often shrouded in silence, with some health-care providers unwilling to deliver safe services. Religion often influences policy and legal reform. Religious values sometimes created shame and stigma, especially if pregnancy occurred out- side of marriage, catapulting women toward seeking an abortion. Some interpreted maternal death, as “appropriate retribution” for both “sins”. Religious teaching in Bangladesh did not challenge the practice of MR, as interventions were permitted “prior to ensoulment” of the fetus, while second-trimester abortion would not be allowed (37). No review of the 1860 British law occurred and no effort was made to legalize all abortion. By promoting MR for contraceptive failure, policy-makers, avoided second-trimester abortion, and the possible public backlash from conservative elements for the MR programme (33). Other highlighted challenges included gender inequality, which limits access to education for girls; the requirement in some settings for spousal approval to use contraceptives, terminate a pregnancy or seek medical care; and girls being expelled from school when they become preg- nant. Stages 2 and 3: country-specific follow-up plans and activities Stage 2 is the development and testing of appropriate interventions. Evaluation must be built into this stage, to ensure that interventions are feasible, acceptable, effective and sustainable and merit scaling up. Table 3 summarizes stage 2 and 3 activities for the five African countries. Common recommendations included development of national guidelines for abortion care, dissemination of information on legal indications for abortion, strengthening FP programmes, and improving adolescent SRH services and sexuality education. In the Republic of Moldova, HRP External Evaluation 2008−2012 Chapter 4 Page | 132 three pilot centres were established to provide CAC and train providers in use of the newly developed guidelines and strategies. During stage 3, these centres were designated as training sites. Table 3 Stages 2 and 3: strategic approach, five African countries Ghana Guinea Malawi Senegal Zambia Law: relatively liberal Law: some restrictions Law: very restricted Law: very restricted Law: relatively liberal Stage 2: strategic development/pilot intervention Develop CAC ser- vices: • National standards, guidelines and pro- tocols (2004, 2006) • Update health in- formation systems to monitor and eval- uate abortion- related services • Train mid-level staff to provide abortion using MVA up to 12 weeks’ gestation Prevention of unwant- ed pregnancy/unsafe abortion included in Strategic plans for safe motherhood (2001– 2010); family planning (2010); health and development of ado- lescents/young people (2008–2012). Areas of focus include: • FP services • commodity security • sexual health educa- tion • improve PAC Develop national standards and guide- lines for provision of abortion MVA equipment added to Standard equipment list (2009); availability limited • Advocacy for legal reform initiated with creation of Coalition for Prevention of Un- safe Abortion (COPUA) • Media sensitization • Wide discussion of discordance between the Maputo Protocol (36) and the law Medical Association of Malawi to develop clini- cal protocols for CAC Goal: liberalization of current law to include grounds for rape/incest • Draft bill prepared by the Association of Women Lawyers of Senegal based on the Maputo Pro- tocol (36) • Advocacy task force formed to conduct awareness raising among parliamen- tarians, religious leaders, journalists and civil society Development of CAC services • National standards, guidelines and pro- tocols (2009) • CAC provided in 28 facilities in three of eight provinces • Operations research undertaken to train nurse practitioners and clinical officers to deliver medical and surgical abortion services Stage 3: scaling-up • Partner with public, private and nongov- ernmental organiza- tions to expand ser- vices • CAC now available in 60 public and pri- vate facilities, in- cluding 12 hospitals • Incremental scaling- up to 18 more facili- ties; goal is national coverage • Given revisions to the law and lack of knowledge about changes, the team recommended: • develop clear legal guidance for applica- tion of the law • wide dissemination of information on le- gal indications for abortion • Ministry of Health has started to strengthen PAC and CAC as le- gally permitted • Infrastructure and equipment need to be assessed in 20 health centres in the capital, Lilongwe • MVA orientation workshops held • Plans are being developed to im- prove the quality of PAC, FP services and SRH education • In two study prov- inces, additional fa- cilities have been added in partnership with Marie Stopes International Enabling environment • Medabon registered for use in pub- lic/private facilities • Misoprostol added to essential drug list for obstetric/ gynaecological use • Fixed price for abor- tion in public facili- ties • FP commodities in short supply • No legal reform ex- pected • Legal requirement that three physicians approve every abor- tion is a barrier • Medabon registered but use limited to obstetrics/ gynaecology; availa- bility limited HRP External Evaluation 2008−2012 Chapter 4 Page | 133 Ghana Guinea Malawi Senegal Zambia Law: relatively liberal Law: some restrictions Law: very restricted Law: very restricted Law: relatively liberal Status • Services have in- creased • Women still opt to self-induce with misoprostol (PAC is free; safe abortion costs US$35–40) • Lack of donor funds to support follow-up activities delayed implementing action plans • Delivery of CAC seems long way off • Efforts needed to improve access to FP services/commodities • Civil society repre- sentatives, support- ed by The Ministry of Health and Pre- vention and the Min- istry of Gender are advocating for legis- lative change re- garding abortion • Project being scaled up nationally In 2011, in collaboration with HRP the WHO Safe abortion: technical and policy guidance for health systems was adapted for use in Bangladesh (32). These guidelines now require that the uterine size and/or time from the LMP be established as part of a medical examination before the procedure. After the procedure, aspirated uterine contents must be examined to ensure that the uterus has been completely evacuated. If inconclusive, guidelines for further assess- ment are included. Stage 3 is intended to scale up proven interventions by building institutional and programmatic capacity to deliver them on a wide scale. The nature and pace of activities depends on the health system’s readiness for change, the broader political and sociocultural environment and available resources. The Republic of Moldova’s Ministry of Health started conservatively, only allowing outpatient CAC up to 8 weeks from a woman’s LMP. After a year of excellent safe practice, the limit was increased to 10 weeks, and will probably be increased to 12 weeks with further evidence of safety. As the high cost of medical abortion using mifepristone limited its use by poorer women in the Republic of Moldova, the government and an NGO, the Reproduc- tive Health Training Centre, were exploring the registration of Medabon as a low-cost mifepris- tone–misoprostol medical abortion product. Lessons learnt The situational analysis The participatory, field-oriented process of the strategic approach is suited to influencing poli- cy and programmes. Ministry of health involvement is vital to ensuring the integration of find- ings into health policy and programmes. The process enables strengthening of local skills in planning, quantitative and qualitative data collection and analysis, intersectoral collaboration, translating research into policies and programmes, evaluation and scale-up of successful inter- ventions. Teams often need technical support to facilitate discussions, given the sensitive and emotional nature of abortion. Engagement of financial partners early in the process is critical, otherwise implementation may become stalled midstream, leading to loss of momentum. Flexibility and adaptation Each country adapted the strategic assessment tool to national priorities and the local context. Where modifying the abortion law may not be feasible, creative Bangladeshi-type solutions may be possible. Other related determinants that could be addressed without legal reform included adolescent sexuality, violence against women, post-exposure EC for rape victims, “off licence” sale of abortifacients like misoprostol, and managing those entry points so that more acceptable SRH practices develop. HRP External Evaluation 2008−2012 Chapter 4 Page | 134 Impact The Moldovan experience is being replicated in Ukraine. While the Republic of Moldova acknowledges the savings that derive from outpatient abortion care versus inpatient care, in- cluding PAC, they experienced professional tension as individual hospitals lost income. While contraceptive utilization has slowly improved in Romania, especially among rural women where the contraceptive prevalence rate increased from 21% in 1999 to 33% in 2004, better- quality care has been associated with falling abortion-related maternal mortality ratios, from 58 per 100 000 live births when restrictions on abortion were removed in 1999 to below 5 per 100 000 in 2006 (38). Conclusions The strategic approach methodology is a powerful tool to break through the silence of unsafe abortion by enabling diverse stakeholders to inform policies and services. Progress in helping women prevent and manage unwanted pregnancy and unsafe abortion ultimately depends on sustained commitment and leadership from governments, civil society and donors, through all three phases of the strategic-approach process. The experience across varying legal and cul- tural settings indicates that countries need not wait for legal reform to improve care. Even if the law is favourable, it does not guarantee that abortion care will be safe. Bangladesh demonstrated a creative solution for an inhospitable legal environment. As countries develop the enabling environment, there is a need to monitor implementation to ensure that the standard of care is consistent with current knowledge and practice. These experiences demonstrate how political will, coupled with international and NGO tech- nical and financial support, can deliver comprehensive reproductive health services aimed at improving women’s health and maternal morbidity and mortality. Romania, however, shows that without consistent attention to management of the supply chain for FP commodities, maintenance of MVA/EVA (electronic vacuum aspiration) equipment and continued training of new members of the health team, the likelihood of reverting to less safe practices is real (39). Table 4 summarizes a framework where countries can locate their barriers to safe abortion care from the country experiences in the case-study and identify possible areas for interven- tion, even if circumstances are far from ideal. HRP has developed a workable formula capable of reducing the risks of unsafe abortion across a range of settings. HRP External Evaluation 2008−2012 Chapter 4 Page | 135 Table 4 Framework to enable access to comprehensive reproductive health and safe abortion care: the combined experiences of African and eastern European countries Areas for action Legal and cultural behaviour change Gendered atti- tudes and values Skills and compe- tencies of health providers Appropriate envi- ronment to deliver safe abortion care Planning and regulato- ry environment Objec- tives • Harmonize laws with international standards • Reduce stigma and discrimina- tion • Improve SRH literacy • Improve clinical and counselling skills and competencies • Ensure access to care • Provide adequate resources • Enable safe abortion care • Monitor access, safe- ty, quality and out- come Targets • General public • Legislators • Political leaders • Religious leaders • Enable SRH education in schools for boys and girls • Develop SRH educators • Provide public education aimed at dis- pelling local myths and negative atti- tudes • Develop and im- plement regulations so that middle-level providers may de- liver FP/safe abor- tion care • Develop teaching skills and training teams (trainer of trainers: in service; tertiary) • Modify curricula at basic, undergradu- ate and postgradu- ate levels, con- sistent with current knowledge and evi- dence • Enable continuing education for health-care provid- ers • Add required drugs and equipment to essential drugs and equipment list • Upgrade facilities to provide care in a confidential, well- equipped environ- ment • Ensure geographic access • Ensure consistent supply of drugs, equipment and supplies • Maintain and re- place equipment as necessary • Provide pre- and post-abortion coun- selling • Ensure affordable access to FP • Ensure comprehen- sive needs assess- ment, which includes diverse input (strate- gic assessment stage 1) • Ensure adequate planning and identifi- cation of resources to enable a successful pilot stage (stage 2) • Have a clear strategy for how the scale-up will be financed and implemented (stage 3) • Have a clear, inte- grated framework for monitoring safety, quality and outcome a Or creatively avoid a public challenge to the existing law, as was done in Bangladesh. 4.2.6 Controlling sexually transmitted and reproductive tract infections: initiative to eliminate mother-to-child transmission of congenital syphilis Background Untreated syphilis in pregnancy can cause late abortion, stillbirth (25% of cases), prematurity or low birth weight (13%), neonatal deaths (11%) and congenital infection (20%) (40). On-site antenatal screening and same day-treatment can, however, reduce all these adverse out- comes. While most countries have policies for antenatal syphilis screening, implementation has faltered from lack of training, inadequate supply of reagents, long turn-around time be- tween testing and result availability, and limited access to drugs to treat infected individuals. In 2007, WHO launched the global initiative to eliminate congenital syphilis, aimed at achieving the following preventive outcomes: • reduce the overall prevalence of syphilis in the adult population; • deliver integrated SRH programmes; • promote and ensure access to high-quality ANC for all pregnant women; • provide syphilis screening and treatment within ANC services. HRP External Evaluation 2008−2012 Chapter 4 Page | 136 Elimination of congenital syphilis (ECS) is a feasible public health goal, which is more cost effec- tive at preventing stillbirths than any other pregnancy intervention besides comprehensive emergency obstetric care (41). Global antenatal attendance rates are high (81% of women worldwide make at least one visit and 51% make four or more visits); screening tests are low cost (US$0.5–3.0) and technically feasible at the primary care level (rapid test reagents can be stored at room temperature for 9–18 months) (42); and treatment with penicillin is inexpen- sive (US$1.50 per treatment) and already on the essential medicines list of all countries (43). New rapid testing technology (42) can also enable joint syphilis and HIV screening in nearly any ANC setting. Despite these facilitating factors, screening pregnant women for syphilis is often not a priority public health intervention, and the burden of congenital syphilis is underappreci- ated. The WHO strategy promotes early ANC (first visit before 16 weeks), testing of all women at the first antenatal visit, repeat testing in the third trimester and treatment of the partners of in- fected women (43). The main challenge is how to increase early attendance to improve the effectiveness of antenatal treatment. To achieve universal screening, the cost may need to be borne by the state, or those most in need of screening may be least likely to afford the cost of testing (44). The WHO initiative for the global ECS aims to decrease the number of cases of congenital syph- ilis by at least 80% in 10 high-burden countries (Table 5) by 2015; these countries account for over 40% of the global syphilis burden (45). The specific targets of the global initiative are that, by 2015: • at least 90% of pregnant women will be screened for syphilis; • at least 90% of women who are syphilis seropositive will be appropriately treated. Table 5 Potential for syphilis elimination in the 10 high-impact countries Country Estimated births per year (2010) (46) % of women mak- ing 1 and 4 antenatal visits (2006–2010) (46) % of ANC wom- en tested at first visit (2010) (47) Prevalence of syphi- lis in antenatal popu- lation (%) (47) Estimated still- births and (still- birth rate per 1000) (2009) (48) Bangladesh 3 038 000 53 and 23 na 0.6 (2008) 128 500 (36) India 27 165 000 75 and 51 65 0.3 (2010) 605 230 (22) Indonesia 4 372 000 93 and 82 na na 62 290 (15) Kenya 1 529 000 92 and 47 59 1.8 (2010) 34 130 (22) Madagascar 732 000 86 and 49 85 3.4 (2010) 14 590 (21) Mozambique 883 000 92 and 53 68 6.0 (2010) 25 560 (28) Rwanda 438 000 98 and 35 75 1.5 (2010) 9 620 (23) South Africa 1 059 000 97 and 87 75 2.2 (2010) 22 560 (20) United Repub- lic of Tanzania 1 862 000 88 and 43 54 4.2 (2008) 47 550 (26) Zambia 600 000 94 and 60 43 5.3 (2010) 14 380 (26) Total 38 625 000 52–98 and 23–87 43–85 0.6–6.0 (2008–2010) 964 410 (15–36) na, not available. The ECS strategy consists of four critical pillars: I: ensure sustained political commitment and advocacy; II: increase access to, and quality of, maternal and newborn health services; III: screen all pregnant women and treat all those with positive test results; IV: implement surveillance, monitoring and evaluation systems. HRP External Evaluation 2008−2012 Chapter 4 Page | 137 Methods RHR exploits the synergies of collaborative effort across HRP work clusters, across depart- ments in WHO and across United Nations (UN) agencies. Strategies include joint ventures with HIV teams to introduce rapid tests; validate rapid test toolkits that use a single strip to test whole blood for both HIV and syphilis; develop bundled diagnostics for use during the perina- tal/neonatal period by examining which combinations make sense; and find ways to support implementation. Expected barriers include concerns regarding penicillin anaphylaxis and ad- dressing issues of timing so that reinfection has the least impact. HRP provided leadership to synthesize global evidence on congenital syphilis, to develop screening and treatment protocols, implementation tools and guiding principles; and to devel- op surveillance and monitoring systems and guidelines for health-systems strengthening. The evidence has been generated from country research and case-studies of the efficacy and effec- tiveness of screening tests and programme interventions. An ECS website provides ready ac- cess to the most current tools and information (49). Essential programme elements aim to: • address enabling environments through a legislative agenda to ensure supportive laws and policies; • promote appropriate health-seeking behaviours, treatment compliance and sexuality education; • support health-care delivery, including STI case-finding in ANC, partner notification and care; • provide a reliable supply of safe, effective medicines and commodities; • support laboratory services, surveillance, research, training and information networks (50). Advocacy and technology to support the campaign The advocacy strategy aims to provide an enabling environment for implementation, including policy and legal reform, drawing on other successful campaigns. A strategy toolkit provides technical support for screening, case identification, contact tracing and protocols for monitor- ing/surveillance (51). Case identification and treatment protocols Cases should be identified with the use of rapid syphilis tests to screen women at routine an- tenatal clinics at the first visit, and all patients attending STI clinics. Partners are to be invited for care. Syphilis treatment protocols require treatment of pregnant women who test positive, their partners, and asymptomatic infants born to infected women. Infants should be followed up quarterly for the first year of life. A single dose of penicillin is recommended, unless contra- indicated (51). Monitoring outcome/surveillance The monitoring framework encompasses regional, national and local contexts with defined output, outcomes and impact indicators and activities. Core annual ECS data are collected and verified at the country level and submitted to WHO or the Joint United Nations Programme on HIV/AIDS (UNAIDS) through WHO regional offices. Programme impact is measured by the pro- gress toward the target incidence of congenital syphilis (≤0.5 per 1000 live births). HRP External Evaluation 2008−2012 Chapter 4 Page | 138 Progress Table 6 highlights progress in Latin America and the Caribbean (LAC), Africa and Asia. Because of their pre-existing infrastructure, LAC countries were best prepared to embrace the initiative, and adopted a plan of action in 2010. Separate guidelines have been developed for LAC coun- tries and have been integrated into national plans in 22 countries (53). Given its pre-existing health infrastructure and organization, Cuba has made the most progress; however, 11 coun- tries report having achieved target incidence rates for congenital syphilis of less than 0.5 per 1000 live births. In Africa, progress has been slow, as more work needs to be done to develop systems to deliver the services and change the culture of late attendance for ANC. In 2011, the initiative was launched in the Asia-Pacific region, with some success reported in India, Malaysia and Myanmar (52). Table 6 Summary of ECS implementation efforts, by region Assessment Latin America and the Caribbean Africa Asia Rationale/ strategy The region adopted a plan of action in 2010; most countries have de- veloped strategies and plans for both ECS and MTCT of HIV Progress has been limited to the development of country strategic frameworks (e.g. Kenya, South Africa) The Asia Pacific United Nations Prevention of Parent-to-Child Transmission (PPTC) Task Force was launched in August 2011, to lead the initiative Development of screening tools Screening included in routine ANC; challenges reported in obtaining results in timely manner Screening demonstrated to be cost effective (range of cost per DALY saved: US$3.97–10.5) Pregnant women present late for ANC: >6 months’ gestation (e.g. Kenya, South Africa) Variations in antenatal syphilis screening practices; routine screening of all ANC clients done in few countries –Malaysia, Pa- pua New Guinea, Sri Lanka, Thailand (52) Development of guidelines Regional and national plans and clinical guidelines in place in 22 of 48 countries (e.g. Brazil, Cuba, Jamaica) (53) Lack of guidelines for service providers Lack of tracking systems for laboratory results – weak diag- nosis/follow-up systems (e.g. Kenya, South Africa) Monitoring and evaluation guide, fact sheets (12 countries) and costing tool developed (August 2011) Roll-out in field: efficiency Cuba has recorded the greatest success in the elimination initiative. Most countries reported 80% cov- erage on ANC and potential to provide the service; however, the quality of service varies across countries Limited training and supervision of field staff Unavailability and poor manage- ment of drugs and supplies (e.g. Kenya, South Africa) Overall weak implementation (e.g. Ghana) Limited data available on pro- cess issues High rates of first ANC attend- ance – 79% Monitoring effectiveness ECS indicators reported annually; surveillance and health information systems challenged by data quality (e.g. 16/48 countries not reporting indicators); legal and technical frameworks strengthened through WHO regional/country-initiated efforts Weak surveillance, monitoring and evaluation (e.g. Kenya, South Africa) WHO regional offices providing guidance on ECS indicators (e.g. Bangladesh, China, India, Thai- land) Reported reduction in syphilis serology among pregnant women in ANC in India and Myanmar Impact PMTCT coverage – 61% (2010) Syphilis testing coverage – 61% (57) 11 countriesa report a target for incidence of congenital syphilis of ≤0.5 per 1000 live births Antenatal syphilis prevalence: 0.1–0.4% in China, India, Malaysia; 6–7% in Papua New Guinea and Solomon Islands (52) Sustainability Declining political will and funding support Lack of sustained political com- mitment and advocacy (e.g. Kenya, South Africa) Need to scale up PMTCT pro- grammes to reverse the trend in infections in neonates DALY, disability-adjusted life-year. PMTCT, prevention of mother-to-child transmission. a Anguilla, Antigua and Barbuda, the Bahamas, Barbados, Canada, Cuba, Chile, Guadeloupe, Guyana, Panama and United States of America. HRP External Evaluation 2008−2012 Chapter 4 Page | 139 Effectiveness and efficiency The availability of tools enabled regional teams to efficiently begin the process of adaptation to local needs. It was also valuable to have an evidence base that advocates could use to mar- ket the initiative to policy-makers, raise awareness among the public and enable the ECS initia- tive to receive priority political support. Some health teams had to be convinced that the prob- lem should be addressed and that the goal could be achieved. The willingness of development partners like the Global Fund to support the integration of ECS into their HIV portfolio will help to promote the initiative, as addressing both problems simultaneously will improve outcomes (see Table 7). Table 7 Effectiveness: can the ECS objectives be achieved? Strengths Weaknesses Opportunities Threats Africa, Latin America, Asia: tools/guiding principles in use in Africa (Ghana), Asia (Chi- na), Latin America (Brazil) Africa: cost effectiveness of syphilis treatment demonstrat- ed Asia: between 2007 and 2010 among pregnant women: • syphilis prevalence has declined in some countries (e.g. Myanmar, from 3% to 0.7%; India, from 2.3% to 0.7%) • in other countries, syphilis prevalence has stabilized at <1% between 2007 and 2010 (e.g. Bangladesh, Sri Lanka, Thailand) (personal communication, L Newman, Medical Officer, WHO/HRP, 2012) Latin America: Paraguay – 2006–2009, screening for syphilis at first visit increased by 20% (personal communica- tion, L Newman, Medical Of- ficer, WHO/HRP, 2012) Global issues: a large propor- tion of the burden of congenital syphilis exists in only a few (10) countries (46) Africa, Latin America, Asia: • syphilis care is a low priority in many health ministries • policy-/decision-makers in Latin America (Plurinational State of Bolivia) and Africa (Kenya, South Africa) are un- aware of syphilis problem and cost effectiveness of treat- ment • health-care providers (Plurina- tional State of Bolivia, Kenya and South Africa) unaware of consequences • community unaware of dis- ease, results in incomplete treatment (Brazil, China) Latin America: • challenges with data quality (e.g. Uruguay, underreporting of congenital syphilis – 45% of cases reported) (54) • syphilis treatment has de- clined (e.g. Paraguay) along- side 24% reduction in syphilis seropositivity among pregnant women (55) Global issues: • lack of integrated systems for STIs and SRH • diagnosis and treatment of partners for syphilis is one of the least reported indicators Global issues: • innovations in combina- tion prevention strate- gies, integrated pro- gramming, political commitment • community-based sur- veillance using skilled birth attendants and traditional healers to support referrals and report pregnancy com- plications, stillbirths and miscarriages • Global Fund proposals to scale up interven- tions integrating STI and SRH services (e.g. Rwanda, United Repub- lic of Tanzania, Zambia) Global issues: • Declining interest in STIs; declining resource alloca- tion for PMTCT • cultural barriers (i.e. gen- der norms that reduce women’s power to negoti- ate safer sexual inter- course; limited mobility and autonomy to make health decisions; culture of silence concerning SRH needs and issues) • incomplete/inadequate reporting and information systems – reported figures need to be interpreted with caution Factors that help programmes successfully get off the ground efficiently (see Table 8) include having evidence that demonstrates the value of integrating the initiative into existing SRH ac- tivities. Challenges include the persistence of vertical programmes (e.g. ANC, HIV) that are poorly integrated into SRH services, weak mid-level human resources skills, and inefficient lo- gistical support. The strategy of bringing implementation teams together to share experiences helps new programmes understand what works, and demonstrates opportunities for integrat- ing these successes into existing services. HRP External Evaluation 2008−2012 Chapter 4 Page | 140 Table 8 Efficiency: achievement of ECS interventions within stated time frames Strengths Weaknesses Opportunities Threats Africa: • research evidence shows that scaled up STI pro- grammes can work • improved STI management in the United Republic of Tanzania has been shown to reduce HIV incidence by 38% • integration of STI services into SRH programmes im- proves service-provider skills and attitudes Africa, Latin America and Asia: • weak management of human resources (especially mid- level) • weak procurement, logistics and distribution of commodi- ties and supplies • many vertical, disease-specific programmes are poorly inte- grated into SRH services • inadequate surveillance, moni- toring and evaluation systems (50) Africa: • capacity to share les- sons across pro- gramme experiences (e.g. adapting what worked in the United Republic of Tanzania) Global issues: • weak implementation • prevailing cultural and legal barriers • declining funding support • weak political will The way forward: sustainability Technology transfer is needed to help scale up services, especially where middle-level man- agement capacity is weak, particularly in Africa and some Asian countries. Continued strength- ening of laboratory facilities, laboratory information systems, surveillance and programme evaluation will improve programme efficiencies in all countries and improve country capacity to generate more reliable data for planning (53). Cultural and legal barriers, inconsistent fund- ing and weak political will to eliminate syphilis are threats to sustainability. The development and implementation of the range of strategies is a dynamic process that will require sustained support from HRP (see Table 9). A 2011 meeting of six African countries (Central African Republic, Ghana, Madagascar, Mozambique, United Republic of Tanzania and Zambia), selected because of their high disease burden (antenatal syphilis rates of 4.4–7.1%), concluded with participants agreeing to draft national action plans to eliminate MTCT of syphi- lis and HIV in their respective countries. Steps outlined in their draft national action plans in- cluded the need to: • improve access to and the quality of ANC services; • introduce or scale up rapid syphilis and HIV testing (including CD4+ testing); • develop indicators and targets for monitoring and evaluation; • develop costed plans of action; • promote advocacy with partners; • create a supportive national policy environment (personal communication, L Newman, Medical Officer, WHO/HRP, 2012). Table 9 Sustainability: capacity development to sustain ECS activity Skills transfer Innovations Opportunities Threats Africa, Latin America and Asia: • more investment in transfer of skills at country level needs to occur, especially in the areas of management, surveillance, data collection, evaluation skills, laboratory testing, and sensitivity training to reduce discrimination and stigmatiza- tion of STI clients LAC: • strengthening surveillance systems for HIV and syphilis (e.g. Argentina 2010) • Global issues: • need to prioritize expansion of access to diagnostic test- ing (51); improving advoca- cy; involvement of commu- nity groups and NGOs Global issues: • many countries are now aware of the importance of syphilis screening and treatment • syphilis treatment is cost effective Global issues: • resource issues (funding) • cultural barriers (i.e. gender norms) • missed opportunities for HIV and syphilis testing of partners of pregnant women in Argen- tina (53) HRP External Evaluation 2008−2012 Chapter 4 Page | 141 The achievement of the target congenital syphilis incidence of ≤0.5 per 1000 live births in 11 LAC countries (54) provides encouragement that the goal is not impossible. Conclusions The ECS campaign has refocused attention on STIs and is a cost-effective goal. By scaling up interventions that integrate STI and SRH services, WHO has demonstrated to country teams strategies to roll out programmes starting from various levels of readiness of a health system and teams, and moving forward. With relatively high disease burden in LAC, this region was best able to tackle the double goal to eliminate both congenital syphilis and MTCT of HIV; oth- er regions have focused on the ECS component. By focusing on a treatable condition such as syphilis, a reduction in HIV transmission rates is an added benefit. This campaign is an excellent example of the public good created by HRP, given its unique capacity to synthesize new tech- nical development into effective clinical interventions that improve health and survival. Through partnerships with academic institutions, NGOs and other international agencies, high- quality research can be undertaken to inform decision-making about what works, and identify the most cost-effective approaches to solving public health problems, marrying skills in high- income countries to the needs of low- and middle-income countries, while building collabora- tion between low- and middle-income countries to help each other. 5 Conclusions and recommendations 5.1 Summary Although it is relatively small, the HRP team in Geneva is impressive in its capacity to identify and coordinate a large network of investigators, collaborators and experts, drawn from aca- demic and research institutions across all the WHO/UN regions of the world, often at short notice, to address acute, as well as long-term, development challenges. This vast network en- sures that whatever problem arises, the best minds on the planet can be assembled to explore the problem, pore over existing evidence or gather new information (56), or innovate to come up with solutions. Sometimes there are setbacks, as the male hormonal contraceptive trial demonstrated, but the global community can be assured that HRP will always promote safe solutions to problems that may be old and intractable, as well as developing the best approach to addressing new challenges. The cases demonstrate the global value of the Special Programme of Research, Development and Research Training in Human Reproduction, providing insight into the stages from problem identification (unsafe abortion), to problem clarification (the strategic approach), to generating new knowledge (male contraception) and strategies (emergency contraception), to global roll- out of solutions to old problems (ECS), while policing new knowledge (hormonal contracep- tives and HIV) that ensures that advice given to individuals, communities and policy-makers is consistent with the best evidence and available expertise to evaluate this information and re- spond appropriately. HRP has evolved procedures and practices that can provide a solution to most SRH problems. The strategic framework enables clarification of problems and crafting of solutions, even to sensitive problems such as unsafe abortion, and can adapt to a country’s level of readiness to address problems either incrementally or comprehensively. 5.2 Impact Tables 10 and 11 look at possible associations between these interventions and changes in global health indicators. The capacity to achieve the first screen-and-treat ECS target is improv- ing, as global access to ANC has grown across all regions. The challenge, however, will be to HRP External Evaluation 2008−2012 Chapter 4 Page | 142 provide the third-trimester screen outside of south-east Asia and the LAC region, as just half of the world’s mothers make four or more antenatal visits. Monitoring outcomes will also be challenging, as, in the most populous regions of the world, fewer than one in two mothers were attended by a skilled attendant at birth in 2010. The higher observed contribution of women to the stabilizing HIV prevalence rate may be a measurement artefact associated with improving antenatal surveillance. We await the next round of indicators to determine whether new initiatives are needed to address the evolving situation. Table 10 Impact I: changes in Millennium Development Goal indicators, selected WHO regions, 2000– 2010 Impact MDG and other related indicators World Sub-Saharan Africa South Asia South-east Asia Latin America and the Caribbe- an 2010 2000 2010 2000 2010 2000 2010 2000 2010 2000 Maternal health ANC – at least one visit (%) 80 71 77 71 71 54 93 90 96 90 ANC – at least 4 visits (%) na na 46 48 48 27 80 70 88 81 Skilled attendance at birth (%) 66 60 45 44 49 36 74 65 92 82 Maternal mortality ratio 210 320 500 740 220 400 150 240 80 100 Family planning Contraceptive preva- lence rate (%) 63.4 61.5 24.6 18.4 55.6 48.2 62.8 57.1 73.3 70.0 Unmet need for family planning (%) 12.4 12.9 25.4 26.5 15.6 17.8 13.4 15.6 10.4 12.6 Adolescent fertility rate per 1000 population 48.6 50.9 119.5 121.9 46.0 58.5 44.8 40.6 78.9 87.5 HIV/STI control HIV prevalence in adults aged 15–49 years (%) 0.8 0.8 4.8 5.6 0.2 0.3 0.4 0.4 0.4 0.5 Cases in female adults aged 15+ years (%) 50 50 59 58 37 35 33 32 37 35 na, not available. a Source: Adapted from: Statistical annex. The Millennium Development Goals report. New York, United Nations, 2012 (57). HRP External Evaluation 2008−2012 Chapter 4 Page | 143 Table 11 Impact II: changes in safe abortion indicators, selected regions, 2003–2008 World Africa Asia Europe Latin America and the Caribbean 2008 2003 2008 2003 2008 2003 2008 2003 2008 2003 Number of induced abortions (millions) (60) 43.8 41.6 6.4 5.6 27.3 25.9 4.2 4.3 4.4 4.1 Number of unsafe abortions (millions) (58,59) 21.6 19.7 6.2 5.5 10.8 9.8 0.4 0.5 4.2 3.9 % Unsafe abortions (60) 49 47 97 98 40 38 9 11 95 96 Abortion-related maternal deaths (58,59) 47 000 66 500 29 000 35 900 17 000 28 500 90 <60 1100 2000 Abortion mortality ratio per 100 000 live births ) (58,59) 30 50 80 110 20 40 1 1 10 11 Proportion of ma- ternal deaths due to unsafe abortion (%) (58,59) 13 13 14 14 12 12 8 6 12 11 Table 11 indicates that previous improvements in the abortion rate seem to have stalled, with a rising proportion of global abortions considered unsafe. With efforts to improve the quality of abortion services and PAC, the data suggest that abortion-related deaths have decreased. 5.3 Recommendations • A clear policy framework is needed to guide research efforts regarding male repro- ductive health, including the way forward in the development of male contraceptives. Within this framework, it may be necessary to revisit the costs and benefits of male contraception. Currently, while the health risks of a new male contraceptives are like- ly to be borne almost entirely by the user, and the health benefits appear to be largely to the partner, in terms of avoidance of the risks of unwanted pregnancy, there are social and economic benefits which accrue to the family unit. The economic costs of an unwanted pregnancy are often the responsibility of the male partner and needs to be factored into the equation. • Efforts need to be explored to advance the social marketing and promotion of EC and other long-term contraceptives to adolescents in particular, as progress in reducing fertility in this age group has been slow. • To support the goals for safe abortion and ECS, monitoring indicators should be in- cluded in the MDG or other universally accepted country reporting frameworks. • Strategies are needed to effectively measure not only access to EC, but also utilization and impact. Integration of measurement tools into demographic and health surveys and other reproductive health surveys may be a place to start. • The increasing HIV prevalence in the antenatal population needs to be monitored, es- pecially in countries with high HIV prevalence. Revitalization of health-promotion ini- tiatives targeting SRH behaviours among adolescents and youth may be a useful start- ing point. HRP External Evaluation 2008−2012 Chapter 4 Page | 144 References 1. World Health Organization, Department of Reproductive Health and Research. Repro- ductive health strategy to accelerate progress towards the attainment of international development goals and targets: global strategy adopted by the 57th World Health As- sembly. Geneva, World Health Organization, 2004 (http://whqlibdoc.who.int/hq/2004/WHO_RHR_04.8.pdf, accessed 9 November 2012). 2. OECD DAC Network on Development Evaluation. Evaluating development co-operation: summary of key norms and standards, 2nd ed. Paris, Organisation for Economic Co- operation and Development, 2010 (http://www.oecd.org/development/evaluationofdevelopmentprogrammes/dcdndep/4 1612905.pdf , accessed 9 November 2012). 3. World Health Organization. The WHO strategic approach to strengthening sexual and reproductive health policies and programmes. Geneva, World Health Organization, 2007 (WHO/RHR/07.7). 4. Simmons R et al. The strategic approach to contraceptive introduction. Studies in Family Planning, 1997, 28:123–138. 5. Department of Reproductive Health Research, World Health Organization. Medical eligi- bility criteria for contraceptive use, 4th ed. Geneva, World Health Organization, 2009. 6. British Medical Association. British medical book awards (http://bma.org.uk/about-the- bma/bma-library/medical-book-awards, accessed 9 November 2012). 7. Combined hormonal contraceptive use during the postpartum period. Geneva, World Health Organization, 2010 (http://whqlibdoc.who.int/hq/2010/WHO_RHR_10.15_eng.pdf, accessed 9 November 2012). 8. UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP). Biennial Report of HRP 2008–2009. Geneva, World Health Organization, 2010 (http://apps.who.int/iris/bitstream/10665/70346/1/WHO_RHR_HRP_10.09_eng.pdf, accessed 3 December 2012). 9. WHO’s strategic vision in sexual and reproductive health and rights: Business Plan 2010– 2015. Geneva, World Health Organization, 2010 (WHO/RHR/10.10). 10. Nieschlag E. Clinical trials in male hormonal contraception. Contraception, 2010, 82(5):457–470. 11. Meriggiola MC et al. Norethisterone enanthate plus testosterone undecanoate for male contraception: effects of various injection intervals on spermatogenesis, reproductive hormones, testis and prostate. Journal of Clinical Endocrinology and Metabolism, 2005, 90(4):2005–2014. 12. Qoubaitary A et al. Pharmacokinetics of testosterone undecanoate injected alone or in combination with norethisterone enanthate in healthy men. Journal of Andrology, 2006, 27(6):853–867. 13. Male hormonal contraceptive trial ending early. Press release. Arlington, VA, CONRAD, 2011 (http://www.conrad.org/news-pressreleases-63.html, accessed 9 November 2012). 14. 24th meeting of the Policy and Coordination Committee: 16–17 June 2011. Geneva, World Health Organization, 2011:6 (HRP/PCC(24)/2011). 15. Trussell J et al. Emergency contraceptive pills: a simple proposal to reduce unintended pregnancies. Family Planning Perspectives, 1992, 24:269–273. HRP External Evaluation 2008−2012 Chapter 4 Page | 145 16. Governments worldwide put emergency contraception in women’s hands: a global re- view of laws and policies (briefing paper). New York, Center for Reproductive Rights, 2004 (http://reproductiverights.org/sites/default/files/documents/pub_bp_govtswwec.pdf, accessed 9 November 2012). 17. Emergency contraception in the Americas: what is EC? (fact sheet). Washington DC, Pan American Health Organization (http://www.paho.org/English/AD/GE/emergencycontraception.PDF, accessed 9 No- vember 2012). 18. Westley E, Glasier A. Emergency contraception: dispelling the myths and misperception. Bulletin of the World Health Organization, 2010, 88:243–244. 19. The Emergency Contraception Newsletter, 2008, 13(1) (http://www.cecinfo.org/publications/newsletters/NewsletterSummer2008.pdf, ac- cessed 9 November 2012). 20. Kotler P, Lee NR, Rothschild M. Social marketing for public health: an introduction. In: Cheng H, Kotler P, Lee NR, eds. Social marketing for public health: global trends and suc- cess stories. Sudbury, MA, Jones and Bartlett Publishers, 2011:1–2. 21. Revitalizing the emergency contraception agenda: how social marketing and NGOs are expanding access to emergency contraception (ICEC brief). New York, International Con- sortium for Emergency Contraception, 2012. 22. Raine TR et al. Direct access to emergency contraception through pharmacies and effect on unintended pregnancy and STIs: a randomized controlled trial. JAMA, 2005, 293(1):54– 62. 23. Gold MA et al. The effects of advance provision of emergency contraception on adolescent women's sexual and contraceptive behaviors. Journal of Pediatric and Adolescent Gyne- cology, 2004, 17(2):87–96. 24. Westley E, Schwarz EB. Emergency contraception: global challenges, new opportunities. Contraception, 2012, 85:429–431. 25. Landler M. Obama backs limit on sale to teenagers of morning-after pill. New York Times, 8 December 2011 (http://www.nytimes.com/2011/12/09/us/politics/obama-expresses- support-for-plan-b-decision.html?_r=1, accessed 9 November 2012). 26. Heffron R et al; Partners in Prevention HSV/HIV Transmission Study Team. Use of hor- monal contraceptives and risk of HIV-1 transmission: a prospective cohort study. The Lancet Infectious Diseases, 2012, 12(1):19–26. Erratum in: The Lancet Infectious Diseas- es, 2012, 12(2):98. 27. UNAIDS and World Health Organization. HIV and hormonal contraception: frequently asked questions. Geneva, UNAIDS, 2012 (http://www.who.int/reproductivehealth/publications/family_planning/FAQ_HIV_horm onal_contraception.pdf, accessed 9 November 2012). 28. Clarification to WHO note to the media issued on 16 February 2012. WHO upholds guid- ance on hormonal contraception and HIV. Geneva, World Health Organization, 2012 (http://www.who.int/hiv/mediacentre/clarification_16feb2012/en/index.html, accessed 9 November 2012). 29. Rodriguez M, Reeves M, Caughey A. Evaluating the competing risks of HIV acquisition and maternal mortality in Africa: a decision analysis. British Journal of Obstetrics and Gy- naecology, 2012, 119:1067–1073. HRP External Evaluation 2008−2012 Chapter 4 Page | 146 30. Sedge G et al. Induced abortion: incidence and trends worldwide from 1995 to 2008. The Lancet, 2012, 379(9816):625–632. 31. World Health Organization, UNICEF, UNFPA and The World Bank. Trends in maternal mortality: 1990 to 2010. Geneva, World Health Organization, 2011. 32. National menstrual regulation services guidelines. Dhaka, Ministry of Health and Family Welfare, People’s Republic of Bangladesh, 2011. 33. Piet-Pelon NJ. Menstrual regulation impact on reproductive health in Bangladesh, a liter- ature review: final report. Bangladesh, Population Council: Near East Operation’s Re- search and Technical Assistance Project, 1997. 34. Jackson E et al. A strategic assessment of unsafe abortion in Malawi. Reproductive Health Matters, 2011, 19(37):133–143. 35. Ministry of Health (Ukraine), WHO, National Medical Academy of Postgraduate Educa- tion et al. Comprehensive Care for Unwanted Pregnancies Project. Final report, Phase 1: August 2009–October 2011. Kyiv, Women Health & Family Planning, 2011. 36. Protocol to The African Charter on Human and People’s Rights on the Rights of Women in Africa. Adis Ababa, African Union, 2003 (http://www.africa-union.org/root/au/Documents/Treaties/Text/Protocol% 20on%20the%20Rights%20of%20Women.pdf, accessed 6 November 2012). 37. Dixon-Muller R. Innovations in reproductive health care: menstrual regulation policies and programs in Bangladesh. Studies in Family Planning, 1988, 20 (2):129–140. 38. Johnson BR, Horga M, Fajans P. A strategic assessment of abortion and contraception in Romania. Reproductive Health Matters, 2004, 12(24 Suppl.):184–194. 39. Benson J, Andersen K, Samandari G. Reductions in abortion-related mortality following policy reform: evidence from Romania, South Africa and Bangladesh. Reproductive Health, 2011, 8:39. 40. Hawkes S et al. Effectiveness of interventions to improve screening for syphilis in preg- nancy: a systematic review and meta-analysis. The Lancet Infectious Diseases, 2011, 11: 684–691. 41. Bhutta ZA et al. Lancet's Stillbirths Series Steering Committee. Stillbirths: what difference can we make and at what cost? The Lancet, 2011, 377(9776):1523–1538. 42. Peeling RW et al. Rapid tests for sexually transmitted infections (STIs): the way forward. Sexually Transmitted Infections, 2006, 82(Suppl. 5):v1–6. 43. The global elimination of congenital syphilis: rationale and strategy for action. Geneva, World Health Organization, 2007. 44. Cheng JQ et al. Syphilis screening and intervention in 500,000 pregnant women in Shen- zhen, the People's Republic of China. Sexually Transmitted Infections, 2007, 83(5):347– 350. 45. World Health Organization. Advancing MDGs 4, 5 and 6: impact of congenital syphilis elimination. Geneva, World Health Organization, 2010 (WHO/RHR/HRP/10.01). 46. The state of the world’s children 2012. Children in an urban world. New York, UNICEF, 2012 (http://www.unicef.org/sowc2012/pdfs/SOWC%202012- Main%20Report_EN_13Mar2012.pdf, accessed 9 November 2012). 47. Scaling up HIV services for women and children: towards eliminating mother-to-child transmission and improving maternal and child health in the context of HIV. In: WHO, UNICEF, UNAIDS. Global HIV AIDS response – epidemic update and health sector progress towards universal access. Progress report 2011. Geneva, World Health Organization, 2011: Table 7.2 page 147 HRP External Evaluation 2008−2012 Chapter 4 Page | 147 (http://whqlibdoc.who.int/publications/2011/9789241502986_eng.pdf, accessed 3 De- cember 2012). 48. Cousens S et al. National, regional, and worldwide estimates of stillbirth rates in 2009 with trends since 1995: a systematic analysis. The Lancet, 2011, 377:1319–1330. 49. World Health Organization. Sexual and reproductive health. Elimination of congenital syphilis (http://www.who.int/reproductivehealth/topics/rtis/cs_global_updates/en/index.html, accessed 2 December 2012). 50. Global strategy for the prevention and control of sexually transmitted infections: 2006– 2015. Breaking the chain of transmission. Geneva, World Health Organization, 2007. 51. The rapid syphilis test toolkit: a guide to planning, management and implementation. London, London School of Hygiene and Tropical Medicine, 2011 (http://www.lshtm.ac.uk/itd/crd/research/rapidsyphilistoolkit/rapid_syphilis_test_toolki t.pdf, accessed 9 November 2012). 52. The Asia-Pacific Prevention of Parent-to-Child Transmission Task Force. Elimination of Parent-to-Child Transmission (http://www.eptctasiapacific.org/, accessed 9 November 2012). 53. Regional initiative for elimination of mother-to-child transmission of HIV and congenital syphilis in Latin and America and the Caribbean: concept document for the Caribbean. Washington, DC, Pan American Health Organization, 2010 (http://www2.paho.org/hq/dmdocuments/2010/Regional%20Initiative%20for%20Elimin ation%20Concept%20Document%20for%20the%20Caribbean.pdf , accessed 9 Novem- ber 2012). 54. 2010 Situation analysis – elimination of mother-to-child transmission of HIV and congeni- tal syphilis in the Americas. Washington DC, Pan American Health Organization, 2011 55. Methods for surveillance and monitoring of congenital syphilis elimination within existing systems. Geneva, World Health Organization, 2011. 56. Wiysonge CS, Hussey GD. Use of systematic reviews in WHO recommendations. The Lancet, 2011, 377(9782):2006. 57. Statistical annex. The Millennium Development Goals report. New York, United Nations, 2012 (http://mdgs.un.org/unsd/mdg/Host.aspx?Content=Data/Trends.htm, accessed 6 November 2012). 58. Unsafe abortion: global and regional estimates of the incidence of unsafe abortion and associated mortality in 2008, 6th ed. Geneva, World Health Organization, 2011. 59. Unsafe abortion: global and regional estimates of the incidence of unsafe abortion and associated mortality in 2008, 5th ed. Geneva, World Health Organization, 2007. 60. Ahman E, Shah IH. New estimates and trends regarding unsafe abortion mortality. Inter- national Journal of Gynecology and Obstetrics, 2011, 115:121–126. HRP External Evaluation 2008−2012 Chapter 4 Page | 148 Annex 1: List of persons interviewed and methods of contact Area Person contacted Method of communication Promoting family planning Mario Festin, Lead Specialist (WHO) Moazzam Ali, Medical Officer (WHO) Mary Gaffield (WHO) Douglas Colvard (CONRAD) Face-to-face interview; e-mail Face-to-face interview E-mail E-mail Preventing unafe abortion Bela Galantra, Lead Specialist (WHO) Nathalie Kapp, Medical Officer (WHO) Face-to-face interview; e-mail Face-to-face interview; e-mail Controlling sexually transmitted and reproductive tract infections Lori Newman, Medical Officer Nathalie Broutet, Medical Officer Face-to-face interview Face-to-face interview HRP External Evaluation 2008−2012 Chapter 4 Page | 149 Annex 2: Revised guidance: combined hormonal contraceptive use during the postpartum period 26 January 2010, Geneva, Switzerland9 Recommendation 1. Consultation participants determined that current WHO guidance regarding the use of combined hormonal contraceptives (CHCs) in non-lactating postpartum women was dis- cordant with the above evidence. The guidance inadequately reflected the gradually de- clining risk of VTE during the postpartum period, and the potential impact of multiple risk factors on VTE formation during this period. 2. In order to more closely fit the available data, WHO has revised their recommendations, stratifying guidance by the time since delivery (<21 days or ≥21 days after delivery), and the presence or absence of additional risk factors for VTE (including previous VTE, throm- bophilia, immobility, transfusion at delivery, postpartum haemorrhage, body mass index >30 kg/m2, pre-eclampsia and smoking, or immediately after a caesarean delivery). WHO discussed the impact of these additional risk factors for VTE on overall VTE risk, as well as return to fertility in non-lactating postpartum women. These revised recommendations are summarized below. 3. Four categories of risk associated with CHC use were outlined: − category 1 – no restriction on use of contraceptive method; − category 2 – advantages of the method generally outweigh the risks; − category 3 – the risks of contraceptive use usually outweigh the advantages; − category 4 – unacceptable health risk if the contraceptive method is used. 4. Prior to 21 days postpartum, the risks of CHC use generally outweigh the advantages and CHCs should generally not be used (category 3); for some women with additional risk fac- tors for VTE other than being postpartum, CHCs should not be used (category 3/4). Wom- en who are breastfeeding during the first 21 days postpartum should not use CHCs (cate- gory 4). 5. Between 21 and 42 days postpartum, the advantages of CHC use generally outweigh the risks and CHCs generally can be used (category 2), although for some women with ad- ditional risk factors for VTE, these methods should not be used unless other more appro- priate methods are not available or acceptable (category 2/3). 6. For women up to 42 days postpartum with other risk factors for VTE, as outlined above, use of CHCs may pose an additional increased risk of VTE. Women’s risk should be as- sessed according to the number, severity and combination of VTE risk factors present. Be- cause each women is unique with respect to her personal risk profile, clinical judgment will be necessary to determine if she may safely use CHCs. Women who are breastfeeding be- tween 21 and 42 days postpartum should not use CHCs (category 4). 7. Finally, in non-lactating women beyond 42 days postpartum, CHCs may be used without restriction. Although the risk of VTE is the same in breastfeeding as non-breastfeeding women, use of CHCs is generally not recommended prior to 6 months postpartum in women who are breastfeeding. 9 These recommendations remain valid until 2012, when RHR will initiate a review of the Medical eligibility criteria for contraceptive use (5). Chapter 4 Page | 150 Annex 3: GRADE evidence profiles – systematic reviews of hormonal contraception use and HIV risk A. GRADE evidence profile: hormonal contraception for women at high risk of HIV Use of hormonal contraception in women who are HIV negative Outcome Type/number of studies (number of participants) Limitations Inconsistency Imprecision Indirectness Quality Estimate of effect Injectable hormonal contraceptive use versus non-use HIV acquisi- tion 8 cohort studies meeting mini- mum quality criteria (26 512) Serious limita- tions Serious incon- sistency No serious imprecision No indirect- ness Low HR range 0.84–2.2: 3 studies increased risk (HR range 1.5–2.2) 3 studies trend towards decreased risk (HR range 0.84–0.94) 2 studies no clear effect (norethisteronea and DMPA reported separately; no clear association; opposite effects for each type of hormonal con- traceptive) Oral hormonal contraceptive use versus non‐use HIV acquisi- tion 7 cohort studies meeting mini- mum quality criteria (25 518) Serious limita- tions Serious incon- sistency No serious imprecision No indirect- ness Low HR range 0.65–1.8: 1 study increased risk (HR 1.5, 95% CI 1.0 to 2.1) 3 studies trend towards increased risk (HR range 1.1–1.8) 3 studies trend towards decreased risk (HR range 0.65–0.91) Note: Publication bias was not formally assessed. Estimates are based on adjusted risk estimates, results from marginal structural model analysis were used when available. CI, confidence interval; HR, hazard ratio. a Three studies that reported risk estimates separately for norethisterone reported no statistically significant effect. Source: http://www.who.int/reproductivehealth/topics/family_planning/hc_hiv/en/ Chapter 4 Page | 151 B. GRADE evidence profile: risk of HIV transmission among women living with HIV and using hormonal contraceptives Use of hormonal contraception in women who are HIV positive Outcome Type/ number of studies (number of participants) Limitations Inconsistency Imprecision Indirectness Quality Estimate of effecta Injectable hormonal contraceptive use versus non‐use HIV transmis- sion 1 cohort study (2476) Serious limi- tations Not applicable (1 study) Serious impre- cision No indirect- ness Low HR = 3.0 (95% CI 1.5 to 6.2) Oral hormonal contraceptive use versus non‐use HIV transmis- sion 1 cohort study (2476) Serious limi- tations Not applicable (1 study) Very serious imprecision No indirect- ness Very low HR = 2.4 (95% CI 0.79 to 7.0) Note: Publication bias was not formally assessed. Estimates are based on adjusted risk estimates, results from marginal structural model analysis were used when available. CI, confidence interval; HR, hazard ratio. a Combined estimate for injectable or oral hormonal contraceptive use versus non-use: HR 2.0 (95% CI 1.1 to 3.7); absolute increase about 1 transmission per 100 person-years. Source: http://www.who.int/reproductivehealth/topics/family_planning/hc_hiv/en/ Chapter 4 Page | 152 C. GRADE evidence profile: HIV disease progression among women living with HIV and using hormonal contraception Use of hormonal contraception in women who are HIV positive Outcome Type/number of studies (number of participants) Limitations Inconsistency Imprecision Indirectness Quality Estimate of effect Hormonal contraception (oral or injectable) versus copper IUD Mortality 1 RCT (599) Serious limitations (1 fair‐quality) Not applicable (one study) Serious im-precision No indirectness Low HR 1.4 (95% CI 0.7 to 3.0);a absolute risk increase 0.88 per 100 woman‐years Progression to CD4 count <200cells/mm3 1 RCT (599) Serious limitations (1 fair‐quality) Not applicable (one study) Serious im-precision No indirectness Low HR 1.6 (95% CI 1.0 to 2.3);a absolute risk increase 3.7 per 100 woman‐years Mortality or progression to CD4 count <200cells/mm3 1 RCT (599) Serious limitations (1 fair‐quality) Not applicable (one study) Serious im-precision No indirectness Low HR 1.6 (95% CI 1.1 to 2.3)a Injectable hormonal contraceptive use versus non‐use Mortality 4 cohort studies (4399) Serious limitations (1 good‐, 2 fair‐, and 1 poor‐quality) No serious incon-sistency No serious imprecision No indirectness Low HR range 0.41–1.0 (no estimate showed statistically significant effect) Progression to AIDS or initia- tion of ART‡ 3 cohort studies (3716) Serious limitations (1 good‐, 1 fair‐, and 1 poor‐quality) No serious incon-sistency No serious imprecision No indirectness Low HR range 0.7–1.0 (no estimate showed statistically significant effect) Mortality, progression to AIDS or initiation of ART‡ 3 cohort studies (4044) Serious limitations (2 good‐, 1 fair‐quality) No serious incon-sistency No serious imprecision No indirectness Low HR range 0.72–1.0 (no estimate showed statistically significant effect) Oral hormonal contraceptive use versus non‐use Mortality 4 cohort studies (1695) Serious limitations (1 good, 1 fair‐, and 2 poor‐quality) No serious incon-sistency Serious im-precision No indirectness Low HR range 0.28–1.1 (no estimate showed statistically significant difference) Progression to AIDS or initia- tion of ART 4 cohort studies (4784) Serious limitations (1 good‐, 1 fair‐, and 2 poor‐quality) No serious incon-sistency No serious imprecision No indirectness Low HR range 0.9– 1.3 (no estimate showed statistically significant effect) Mortality, progression to AIDS or initiation of ART 3 studies (4680) Serious limitations (2 good‐, 1 fair‐quality) No serious incon-sistency No serious imprecision No indirectness Low HR range 0.65–1.0 (no estimate showed statistically significant effect) Note: Publication bias was not formally assessed. Estimates are based on adjusted risk estimates, results from marginal structural model analysis were used when available. ART, antiretroviral therapy; CI, confidence interval; HR, hazard ratio; RCT, randomized controlled trial. a Point estimates were higher for DMPA versus IUD compared with oral contraception versus IUD but confidence intervals were wide and overlapping Source: http://www.who.int/reproductivehealth/topics/family_planning/hc_hiv/en/ Page | 153 Chapter 5 Research-capacity strengthening and network building Chisale Mhango Senior Lecturer, Department of Obstetrics and Gynaecology, College of Medicine, Blantyre, Malawi Page | 154 Page | 155 Contents Abstract . 157 Introduction . 157 Methods . 157 Findings . 157 Conclusions . 161 1 Introduction . 163 1.1 HRP’s strategy for capacity building for research . 163 1.2 Regional advisory panels . 164 1.3 Objectives of the evaluation for research capacity-strengthening and network building. 165 1.4 Scope of the evaluation . 165 2 Methods . 165 3 Findings . 166 3.1 Overview . 166 3.2 Relevance and effectiveness in fulfilment of HRP’s objectives . 166 3.3 Development of regional networks . 169 3.4 Outputs of HRP-strengthened research centres are being used to improve health and save lives . 171 3.5 Sustainability of HRP’s work . 171 3.6 HRP work through research capacity strengthening and network building has impacted on women’s reproductive health and children’s health globally . 172 3.7 The extent to which HRP contribute to global goods . 174 3.8 The extent to which capacity-strengthening and network-building outputs have reflected the mandate of HRP. 174 3.9 The comparative advantage of HRP for research-capacity strengthening and network building and as global leader for research in human reproduction . 174 3.10 The extent to which HRP work has been relevant to programme countries, particularly to low income countries. 175 3.11 The extent to which HRP’s work in research-capacity strengthening is contributing to the achievement of MDGs, ICPD agenda, poverty reduction and women’s health in general . 176 3.12 Programme strengths and weaknesses . 176 4 Conclusions . 176 5 Lessons learnt . 177 6 Recommendations . 177 References . 180 Annex 1: Centre recipients of long-term institutional development grants from HRP that were sent a questionnaire by the external evaluator . 182 Page | 156 Abbreviations CENEP Centro de Estudios de Población [Centre for Population Studies] CIR competitive intraregional research CREP Centro Rosarino de Estudios Perinatales [Centre for Perinatal Studies] CWS courses, workshops and seminars EmONC emergency obstetric and neonatal care HRP UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction HVH Hung Vuong Hospital (Viet Nam) IByME Instituto de Biología y Medicina Experimental [Institute of Experimental Biology and Medicine] ICPD International Conference on Population and Development ICPD/PoA Programme of Action of the International Conference on Population and Development LID long-term institutional development MDG Millennium Development Goal MTCT mother-to-child transmission NIH National Institutes of Health OWER organization-wide expected result PCC Policy and Coordination Committee RAP regional advisory panel RCS research-capacity strengthening REG re-entry grant ReproNet-Africa African Network for Research and Training in Sexual and Reproductive Health and HIV RHR WHO Department of Reproductive Health and Research RMC resource maintenance and capital RMG resource maintenance grant RPM research project monitoring RTG research training grant SGC service guidance centre SRH sexual and reproductive health SSG small supplies grant STAG Scientific and Technical Advisory Group STI sexually transmitted infection UNDP United Nations Development Programme UNICEF United Nations Children’s Fund UNFPA United Nations Population Fund VIA visual inspection with acetic acid WHO World Health Organization HRP External Evaluation 2008−2012 Chapter 5 Page | 157 Abstract Introduction The World Health Organization’s (WHO’s) Global reproductive health strategy includes supporting action-oriented research and research-capacity strengthening (RCS) that contribute to the overarching goal of achieving universal access to quality sexual and reproductive health (SRH) services. The United Nations Development Programme (UNDP)/United Nations Population Fund (UNFPA)/ United Nations Children’s Fund (UNICEF)/World Health Organization (WHO)/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP) is the research arm of WHO’s Department of Reproductive Health and Research (RHR). The goal of HRP in RCS is “to improve reproductive health in countries and regions through support to priority national/regional research, in particular that which is linked to improved operations of reproductive health programmes”. The strategies to achieve a sustainable local resource for national governments and countries of evidence on policy and programme interventions to advance SRH include provision of various forms of grants, including long-term institutional development (LID) grants to institutions to develop the infrastructure for research. After the period of the LID grants, institutions are monitored and become eligible for resource maintenance and capital (RMC) grants to sustain the gains made. HRP has established and convenes regional advisory panels (RAPs) that serve as regional scientific and technical advisory bodies to HRP on priority national/regional research, capacity- building and programmatic activities in SRH. Methods The methodology and process of this evaluation involved desk review of relevant documents related to HRP’s work during the period of the evaluation. A global e-mail questionnaire was also sent to 25 institutions that have benefited from LID grants. The objective of the questionnaire was to assess the performance of the collaborating centres in relation to the support given by HRP. Site visits were made for verification of reports and to assess the extent of national and regional outcomes. Discussions were also held with the ministries of health and other national institutions that collaborated with the collaborating centres, to assess how the collaborating institutions are impacting on these institutions. Findings The findings presented are based on the desk reviews, analysis of the responses from the global questionnaire and the site visit. Overview HRP was funding and collaborating with 103 research centres in 55 countries around the world (11 in the Region of the Americas, 10 in the European Region, 23 in the African and Eastern Mediterranean Regions, and 11 in the South-East Asia and Western Pacific Regions). The questionnaire was sent to 25 LID grant recipients in the Americas, Africa, Asia and the Pacific, and Eastern Mediterranean. Twenty-three heads of the collaborating institutions in 23 institutions responded (5 in Latin America; 10 in Africa; 7 in Asia and the Pacific, and 1 in the Eastern Mediterranean), which represents a 92% response. HRP assists the institutions to identify their needs and visits the sites a number of times to provide expertise on the process of implementation of the agreed activities. Sixty-four per cent of the centres felt that these HRP visits were frequent enough to provide the support that the centres needed. The visits HRP External Evaluation 2008−2012 Chapter 5 Page | 158 also encouraged the institutions to adopt corrective interventions for any shortcomings, to further improve the research orientation. Relevance and effectiveness in fulfilment of HRP’s objectives The extent to which HRP objectives related to capacity building for research were still valid can be deduced from the fact that, in most countries, heads of the HRP-supported centres sit on bodies that set the national research agenda for SRH, and WHO/HRP guidelines and handbooks for SRH programmes are universally used to guide national programmes. In most institutions, the number of courses offered by the centres and the number of presentations at scientific meetings increased after the HRP support. Development of regional networks Through network building, HRP has promoted interregional collaboration in promotion of women’s and children’s health and development of global public goods. Although, understandably, emphasis has been on support for low- and middle-income countries, collaboration with high-income countries has led to development of global goods such as methods for medical abortion. Researchers feel the networks are very useful for exchanging views and learning. Eighty-nine per cent of the centres collaborated with other institutions: of these, 63% collaborated with other local institutions, thereby cascading the knowledge and skills benefit from HRP support, while 37% collaborated with regional institutions and with global/international institutions. In some countries, this collaboration has enhanced the capacity to attract funding from other institutions. For example the Centro Rosarino de Estudios Perinatales [Centre for Perinatal Studies] in Argentina has been able to attract diverse funding towards its activities. This also promotes HRP objectives to provide evidence-based practice in the delivery and provision of clinical reproductive health services, thereby accelerating the achievement of Millennium Development Goals (MDGs). Outputs of HRP-strengthened research centres are being used to improve health and save lives There are many examples of research results that have led to improvement in women’s lives. For example, a study that concluded that controlled cord traction can be omitted with little increase in the risk of postpartum haemorrhage in settings where skilled birth attendants are not available, has potential to save many lives in low- and middle-income countries where postpartum haemorrhage remains the top cause of maternal death. A multicountry survey on maternal and newborn health, with a focus on the management of severe complications in pregnancy and childbirth, in 29 countries, has the potential to reduce case-fatality rates from obstetric complications and save many lives. Sustainability of HRP’s work From the start, the supported institutions are encouraged to develop a sustainability plan to allow them to continue thriving beyond the period of support. A number of institutions have since been weaned off HRP support, in many countries, including Argentina, Brazil, Kenya, Senegal, Tunisia and Zimbabwe, but have continued to turn out large quantities of good- quality research results. However, the ending of the LID grant has led to the loss of salaries for some research staff in some centres, which has resulted in their leaving, with consequent weakening of the centres. This has reduced the capacity of centres to train others. Other services that have suffered are library services and journal acquisition, which have become depleted. HRP External Evaluation 2008−2012 Chapter 5 Page | 159 HRP work through research-capacity strengthening and network building has impacted on women’s reproductive health and children’s health globally Among recorded policy changes and practice are the use of emergency contraception and provision of safe medical abortion care in many countries worldwide; adoption of the use of magnesium sulfate in the management of pregnancy-induced hypertension; use of misoprostol in the management of postpartum haemorrhage and incomplete abortion; and replacement of dilatation and evacuation with manual vacuum aspiration. Together, these have the potential to significantly reduce maternal mortality and morbidity and to create significant cost savings in the delivery of health care. Although studies have not been conducted to assess the impact of centres on health-related MDGs in their countries, there is evidence that work by supported centres has contributed to the achievement of MDGs in their countries. For example in Côte d’Ivoire, the results of the Emergency Obstetric and Neonatal Care survey made the Government introduce the delivery of free care for children and pregnant women, thereby increasing coverage, which will accelerate the achievement of MDGs 4 and 5 in Côte d'Ivoire. In Viet Nam, Hung Vuong Hospital collaborated with WHO task forces to investigate the effectiveness of several interventions, in an effort to minimize maternal morbidities and mortalities through calcium supplementation or supplementation with vitamins C and E for prevention of pre-eclampsia, or misoprostol for prevention and treatment of postpartum haemorrhage. In Peru, a study demonstrated that burning biofuel led to reductions in birth weight over and above the effect produced by living in a hypoxic environment. Further studies led to the introduction of biofuel kitchens in hypoxic high-altitude areas, which reduces the incidence of low birth weight. It is estimated that the use of 500 000 such improved kitchens may reduce the rates of low birth weight. The centres have also contributed to the promotion of an International Conference on Population and Development (ICPD) Programme of Action (PoA) in their countries. For example, in Ethiopia some of the studies that were conducted by the centre were on family planning and abortion; these are ICPD/PoA studies. In Kenya, adoption of family planning guidelines and promotion of contraception among women who are HIV positive has promoted the ICPD agenda. In the Plurinational State of Bolivia, as a result of the institutional research agenda, new topics have been introduced in population studies, such as, interculturality, violence against women, teenage pregnancy and motherhood. Implementation research is increasing, but rather slowly, probably because of inadequate funding. On discovering that the country could not afford to scale up the Papanicolaou smear test for cervical cancer screening, the United Republic of Tanzania participated in a six-country (ReproNet) visual inspection with acetic acid (VIA) study, which found that VIA was acceptable and feasible. VIA has since been adopted in these countries, leading to scaling up of screening services for cervical cancer, the commonest cancer in women in low- and middle-income countries. The extent to which HRP contributes to global goods An example of global goods from HRP-supported research is the non-scalpel vasectomy developed by the Sichuan Family Planning Research Institute in Chengdu, China. This method is now practised worldwide and has made vasectomy more acceptable and accessible to many individuals. Research on the development of the award-winning Odon device, which can potentially be safely used by nurses in rural health centres without risk of maternal and newborn trauma, is another global good that could provide a low-cost simplified way to shorten the second stage of labour without recourse to caesarean section. HRP External Evaluation 2008−2012 Chapter 5 Page | 160 The extent to which capacity-strengthening and network-building outputs have reflected the mandate of HRP The institutions that have benefited from RCS have subsequently been able to participate in implementation of the global research agenda. HRP has been very effective in developing the research capacity of institutions, leading to increased high-quality output from the collaborating centres published in peer-reviewed journals. There are many examples that reveal that supported institutions have conducted studies that influenced national health policy, and that in turn led to improvement in the health of women in all regions. HRP support has enabled researchers in low- and middle-income countries to undertake and manage new types of projects and take advantage of previous research outputs to undertake in-depth studies. Satisfaction among the countries with HRP support is high; 74% of the centres surveyed indicated that HRP had fully met their expectations. Through training and provision of logistics support by HRP, a culture of research has been established in many low- and middle-income countries. The comparative advantage of HRP for research-capacity strengthening and network building and as a global leader for research in human reproduction WHO remains the organization that countries look up to for guidance on health promotion, and countries are more likely to adopt HRP guidelines, which are informed by HRP-supported studies globally, than those produced by other partners. There are a number of other players in the area of research capacity-development in reproductive health in low- and middle- income countries, but they are mostly either supporting or complementing HRP and not competing with it, as HRP is seen to be better placed to influence policy and programmes. Studies funded by HRP, by being focused on country needs, have led to the development of treatment guidelines and global standards that are used in SRH programmes worldwide, focusing on the five key causes of maternal mortality, notably postpartum haemorrhage, infection, eclampsia, obstructed labour and complications resulting from unsafe abortion. Among these are the award-winning Medical eligibility criteria for contraceptive use; Safe abortion: technical and policy guidance for health systems; and guidelines on antenatal care, postpartum haemorrhage, and birth spacing. Seventy-five per cent of the institutions felt that this advantage was because HRP work was more specific to SRH research issues; 45% also felt that this was related to the high-quality training provided through sponsorship to good institutions and the provision of experts for local training. One third reported that HRP promoted studies that were more specific to national needs, which was greatly appreciated by the countries. The extent to which HRP work has been relevant to programme countries, particularly to low-income countries HRP outputs have been accessible to all and used to improve lives. Studies supported by HRP have provided evidence that service coverage can be safely scaled up with task-shifting or sharing, and many low- and middle-income countries have reached more people with essential health services this way. HRP’s RCS initiatives are also influencing the development of women’s and children’s programmes in many countries. Research staff from the collaborating centres sit on national SRH research policy planning bodies, thereby influencing the development of reproductive health programmes in their countries and, in some cases, their region. Among recorded policy changes and practice are the adoption of VIA of the cervix to screen for cervical cancer, to replace the Papanicolaou smear, which was not affordable for most low- and middle-income countries, thereby reducing the prevalence of cervical cancer. HRP External Evaluation 2008−2012 Chapter 5 Page | 161 The extent to which HRP’s work in research capacity-strengthening is contributing to the achievement of MDGs, ICPD agenda, poverty reduction and women’s health in general HRP has funded the training of a large number of scientists in research methodology, as well as postgraduate education for senior researchers, which has influenced policy and programmes in low- and middle-income countries and promoted evidence-based treatment and programming of health care in low- and middle-income countries. This is promoting universal coverage of SRH services, while also improving the quality of care, e.g. through the adoption of focused antenatal care. HRP has been very effective in developing the research capacity of collaborating institutions. The major outputs attributable to HRP in the collaborating centres include an increase in high- quality research proposals and paper output, and the development, or revision, of national guidelines. This is contributing to achievement of national MDGs; countries have also benefited from HRP collaboration through the promotion of the ICPD/PoA. Conclusions There is enough evidence that HRP objectives to strengthen capacity for research are achieved or are likely to be achieved. Testimonials from institutions consider the HRP support as the only real possibility to train with first-line researchers in the area of reproductive health. Workshops and seminars were found to be an enriching experience that create a scientific frame of mind There were few financial opportunities to travel to training centres for most researchers in low- and middle-income countries, other than those provided by HRP. The result is a greatly increased output of publications in peer-reviewed journals that have worldwide influence on policy and practice for the improvement of women’s and children’s health. There is evidence that work by supported centres has contributed to the achievement of MDGs in their countries. In most countries, heads of the HRP collaborating centres sit on bodies that set the national research agenda for SRH, and WHO/HRP guidelines and handbooks for SRH programmes are universally used to guide national programmes. The centres have also contributed to the promotion of ICPD the PoA in their countries. For example in Kenya, adoption of family planning guidelines and promotion of contraception among women who are HIV positive has promoted the ICPD agenda. In the Plurinational State of Bolivia, as a result of institutional research agenda, new topics have been introduced in population studies, such as interculturality, violence against women, teenage pregnancy and motherhood. Implementation research is increasing, but rather slowly, probably because of inadequate funding. The results of a study on the coverage of antenatal syphilis screening and predictors for not being screened in Ulaanbaatar, Mongolia were used to develop an operational research proposal on one-stop service for antenatal syphilis screening, which greatly increased coverage for syphilis screening. A study in the United Republic of Tanzania, which found that VIA was acceptable and feasible, led to the scaling up of screening services for cervical cancer, the commonest cancer in women in low- and middle-income countries. Development of networks to promote the development of global goods, and the promotion of sharing of information and skills in research have been effective in strengthening research capacity and accelerating improvement in women’s health. However, gaps still remain in the needs of individual countries, as a result of inadequate funding to satisfy the needs of the centres. In the face of reduced funding to HRP, it remains to be seen whether reductions in the amounts of RCS grants and lengths of study will result in a slower pace of RCS by HRP. HRP External Evaluation 2008−2012 Chapter 5 Page | 162 Some institutions are still not able to stand on their own after the 10-year LID grant. Many centres in low-income countries will still not have adequate numbers of research staff at the end of the 10-year support for institutional development. This is partly because of staff mobility and partly due to countries’ dependence on HRP support. The ending of LID grants in some institutions has led to the loss of salaries for some research staff, with consequent loss of staff and weakening of the centres. This reduced the capacity of centres to train others. However, these are exceptions rather than the rule, as many institutions received LID grants from other institutions and resource maintenance grants from HRP. WHO remains the organization that countries look up to for guidance on health promotion, and countries are more likely to adopt HRP guidelines that are informed by HRP-supported studies globally than those produced by other partners. HRP External Evaluation 2008−2012 Chapter 5 Page | 163 1 Introduction 1.1 HRP’s strategy for capacity building for research The World Health Organization’s (WHO’s) commitment to attaining global reproductive health goals (Millennium Development Goals (MDGs) 4, 5 and 6), the goals of the 1994 Cairo International Conference on Population and Development (ICPD), and WHO’s Global reproductive health strategy (1), which, among others, underpin HRP’s work, include supporting action-oriented research and research capacity-strengthening (RCS) that contribute to the overarching goal of achieving universal access to, and quality of sexual and reproductive health (SRH) services. Such RCS is premised on the need for self-reliance in generating solutions to national SRH problems and contributes to sustainable development. The United Nations Development Programme (UNDP)/United Nations Population Fund (UNFPA)/United Nations Children’s Fund (UNICEF)/World Health Organization (WHO)/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP) is the research arm of WHO’s Department of Reproductive Health and Research (RHR). The goal of HRP in RCS is “to improve reproductive health in countries and regions through support to priority national/regional research, in particular that which is linked to improved operations of reproductive health programmes” (2). The strategies deployed by HRP aim to ensure a sustainable local resource to national governments and countries for evidence on policy and programme interventions to advance SRH. They include promotion of sharing lessons learnt within and between countries, through network building. Preference is given to “South-to-South” collaboration1, to provide a working environment similar to the one in which the researchers operate. HRP is also using a variety of mechanisms to promote and develop research capacity. These include the provision of individual research training grants (RTGs) to promising researchers to follow research degrees or postgraduate courses in epidemiological, biological biostatistics or social sciences related to SRH. HRP also funds group learning activities that further strengthen research capacity through courses, workshops and seminars (CWS) grants. Long-term institutional development (LID) grants are awarded to institutions to develop their infrastructure for research, hire additional staff critical to conducting research and purchase laboratory equipment, data-management software and hardware and supplies linked to the studies. During the period of the LID grants, institutions are also provided with technical assistance and group learning materials by HRP staff or other regional experts, to address any gaps identified. After the period of the LID grants, institutions are monitored and become eligible for resource maintenance and capital (RMC) grants to sustain the gains made by the research centre following the LID. Institutions are also funded through service guidance centre (SGC) grants to serve at the national level, as a resource for the dissemination, adaptation and adoption of tools and guidelines into health systems and service delivery. Other mechanisms include competitive intraregional research (CIR) grants, small supplies grants (SSGs), and research project monitoring (RPM) grants. For details, please see: http://www.who.int/reproductivehealth/topics/countries/grants/en/index.html. Grants for RCS include support to research proposals that are integrated within the RCS package. Examples of research and other areas of collaboration with centres for capacity building include: 1 Collaboration between countries in the southern hemisphere with similar incomes; these are mostly low- and middle-income countries. HRP External Evaluation 2008−2012 Chapter 5 Page | 164 • development, testing, introduction and application of new methods of family planning, including their promotion within service programmes, through guidance tools to expand choice; • development and testing of methods for dual protection against unplanned pregnancies and sexually transmitted infections (STIs); • development and testing of interventions for prevention of STIs and HIV, including microbicides and protocols or models for the prevention of mother-to-child transmission (MTCT) of HIV; • generating and synthesizing evidence for development and testing of interventions for the promotion of best practices for improving maternal and newborn health; • generating evidence and new knowledge for action, policy, programme support and advocacy for best practices for addressing the SRH of adolescents; • developing action plans for the prevention of unsafe abortion and provision of post- abortion care; • generating evidence and testing standards for the prevention and management of STIs, infertility and cancer of the cervix; • learning users’ and potential users’ perspectives with regard to reproductive health technologies and services; • developing and testing of prevention interventions for unintended pregnancies; • promotion, through research and programme development, of models of best practice for male involvement in reproductive health; • promotion of gender equity and rights in reproductive health, including research and research-linked service and advocacy programmes for the elimination of traditional harmful practices and gender-based violence; • participation in the development and implementation of programmes to support global initiatives for promoting SRH; • facilitation of the dissemination and promotion of utilization of normative guidelines and tools, including advocacy for best practices. Scholarships are also awarded to young investigators to spend an average of 3–6 months in a reputable research institution to perfect their analytical skills, using relevant databases and coursework. This has included attachments to laboratories to develop their expertise in methods and procedures. After receiving individual training, trainees will often return to their original institutions, with projects funded through re-entry grants (REGs) to assist them to reintegrate in their country of origin and put their newly acquired skills into practice. 1.2 Regional advisory panels HRP has established and convenes regional advisory panels (RAPs) that consist of researchers, SRH programme managers, public health specialists and other experts drawn from the region, for primary advice. The RAPs serve as regional scientific and technical advisory bodies to HRP on priority national/regional research, capacity-building and programmatic activities in SRH. They receive and review applications for research. The research protocols are reviewed for relevance and acceptability of the methodologies and funded on approval. LID and related grant applications are also reviewed and funded, based on the findings of a site visit, the needs of the institution and the relevance to the country. RAPs consider the availability of other research institutions operating in the country at the time and the support that the applicants HRP External Evaluation 2008−2012 Chapter 5 Page | 165 get from other funding agencies. RAPs also decide on other activities, such as dissemination of research results and funding of consultants to support specific RCS activities through RPM grants, related seminars and workshops, among others. There are currently four RAPs: one each for the Region of Americas; the African and Eastern Mediterranean Regions; the South-East Asia and Western Pacific Regions; and the European Region, including Central Asian republics. These panels sit at least once a year to review the work of the collaborating centres and advise accordingly. They also undertake missions to the research centres to promote their work and address some of the challenges identified with or without HRP staff. 1.3 Objectives of the evaluation for research capacity-strengthening and network building The objective of this evaluation includes assessment of: • the achievement of HRP objectives and provision of evidence that these objectives can be achieved or that the activities are contributing to the achievement of relevant WHO organization-wide expected results (OWERs, especially for OWERs 4.1, 4.2 and 4.7), and make recommendations on how best to achieve them more efficiently and effectively; • the comparative advantage of HRP in RCS and network building and the impact and sustainability of its work. 1.4 Scope of the evaluation This evaluation assesses how LID grants, research training grants, courses, workshops, seminars and other training support over the period 2008–2012 have assisted in the achievement of HRP objectives in the regions assessed. While there are over 100 institutions collaborating with HRP, receipt of a LID grant signifies a national need for capacity strengthening, and thus the evaluation has concentrated on how recipients of these grants have been strengthened. Emphasis was on support to low-income countries in Asia, Africa and South America. The evaluation has not included assessment of the impact and effect on beneficiaries other than the promotion of processes that are known to lead to improvements in women’s reproductive and children’s health. 2 Methods This exercise involved three stages: 1. desk review of relevant documents, including HRP annual technical reports for 2008–2012; the WHO Global reproductive health strategy (1), RAP reports for 2008–2012, and reports of the Scientific and Technical Advisory Group (STAG) and the Policy and Coordination Committee (PCC). A recent study by interns Oscar Zazueta-Fierro and Alfredo Fort (personal communication, 14 September 2012) on research-capacity training provided additional information to this evaluation; 2. a global e-mail questionnaire was developed to collect relevant data and information. The questionnaire was sent to 25 institutions that have benefited from LID grants; this included eight that received grants before the period of this evaluation, to assess the performance of these institutions after being weaned off the grant (see list in Annex 1). The objective of the questionnaire was to assess the performance of the collaborating centres in relation to the support given by HRP. Indicators included were: HRP External Evaluation 2008−2012 Chapter 5 Page | 166 − the number of people trained within the institution; − research output; − evidence of responsiveness to changing SRH research needs and priorities in the programme of the centres; − the impact of research on national policy formulation; − the impact on health services (adoption and utilization of research findings); − involvement of staff in national, regional and international advisory bodies/technical committees; − the networking role within the countries and also at the regional level; 3. site visits for verification of reports and to assess the extent of national and regional outcomes. Discussions were also held with the ministries of health and other national institutions that collaborated with the collaborating centres, to assess how the collaborating institutions are impacting on these institutions. 3 Findings The findings reported are based on the desk reviews, analysis of the responses from the global questionnaire and the site visits. 3.1 Overview HRP was funding and collaborating with 103 research centres in 55 countries around the world (11 in the Region of the Americas, 10 in the European Region, 23 in the African and Eastern Mediterranean Regions, and 11 in the South-East Asia and Western Pacific Regions). The questionnaire was sent to 25 LID grant recipients in the Region of the Americas, the African Region, South-East Asia and Western Pacific regions, and the Eastern Mediterranean Region; 25 heads of the collaborating institutions in 20 countries responded (5 in Latin America; 10 in Africa; 7 in South-East Asia and the Western Pacific Regions, and 1 in the Eastern Mediterranean), which represents a 92% response. Of these 25 institutions, eight had LID grants before the period of this evaluation and 17 during the period of this evaluation. HRP continues to identify countries that need support for RCS; in 2011, 13 institutions were awarded LID grants. Seven institutions received SGC grants, 7 received CIR grants, 10 received RMC grants and 1 institution received a pre-LID grant. RTGs in the form of courses, workshops, and seminars were awarded to three institutions in the African Region. In the Region of the Americas, 16 fellows received awards for courses or practical training. After identifying the institutions to support, HRP assists them to identify their needs and makes visits to the sites to provide expertise on the process of implementation of the agreed activities. Sixty-four per cent of the centres felt that these HRP visits were frequent enough to provide the support that they needed. The rest wished for more visits. It is important to mention here that HRP repeatedly informed the centres that there were inadequate resources for more frequent visits to the centres. Nonetheless, the visits also encouraged the adoption of corrective actions to any shortcomings noted, to further improve the research orientation in the reproductive health forum of the country (for research needs, prioritization, reviews and translating findings to practice guidelines). 3.2 Relevance and effectiveness in fulfilment of HRP’s objectives The extent to which HRP objectives of the capacity building for research were still valid can be deduced from the fact that the HRP strategy is aligned to the Programme of Action of the HRP External Evaluation 2008−2012 Chapter 5 Page | 167 International Conference on Population and Development (ICPD/PoA) (3), MDGs and the United Nations (UN) Secretary-General’s Global strategy for women’s and children’s health (4). In most countries, heads of the HRP collaborating centres sit on bodies that set the national research agenda for SRH, and WHO/HRP guidelines and handbooks for SRH programmes are universally used to guide national programmes. A recent study by HRP interns Oscar Zazueta-Fierro and Alfredo Fort (personal communication, 14 September 2012) revealed that while there were nine fellows in training 5 years before the LID grants, there were 25 fellows 5 years after the LID grants. Over the same period, the number of courses offered by the centres increased from 12 to 30, presentations at scientific meetings increased from 24 to 95, and publications increased from 5 to 31 per centre. This is proof that the HRP objectives to strengthen capacity for research are being met. As regards consistency between the activities and outputs of the programme in capacity building with the overall goal and the attainment of HRP objectives, it is noted that through the collaborating centres, HRP has trained a large number of people in research methodology and supported postgraduate education to senior researchers who have influenced policy and programmes in their countries. At least 1492 participants benefited from short-term training courses, while 35 had long-term postgraduate training through HRP funding in the 23 institutions studied during the period 2008–2012. The institutions that have benefited from RCS have subsequently been able to participate in implementation of the global research agenda (J Cottingham, Independent Consultant, personal communication, 2012). One expectation of RCS is that institutions will conduct studies that will influence national health policy and lead to improvement in the health of women and all the population. There are many examples that this occurred in all regions. HRP has been very effective in developing the research capacity of institutions, leading to increased high-quality output from the collaborating centres published in peer-reviewed journals that have influenced policy and delivery of care. These include the Centro Rosarino de Estudios Perinatales [Centre for Perinatal Studies] (CREP) in the Region of the Americas, Zimbabwe in the African Region, and Viet Nam in the South-East Asia Region. CREP had 10 publications in peer-reviewed journals in 2010 and 9 in 2011. In 2011 alone, 133 papers were published by collaborating centres globally; in addition, the scope of activities in research institutions has been broadening and expanding, from clinical studies to epidemiological and social science studies; and from descriptive studies to multicentre and operational studies. HRP support has enabled researchers to develop and manage new types of projects and take advantage of previous research outputs to undertake in-depth studies. One quarter of the institutions contacted indicated that they had collaborated with HRP in implementation research. Many countries, such as Afghanistan, revealed increased research proposals and projects in their national tertiary hospitals since their HRP collaboration started. Seventy-four per cent of the centres surveyed indicated that HRP had fully met their expectations The collaborating centres in all the regions are contributing to knowledge, with large numbers of publications in international and national peer-reviewed journals, as shown in Table 1. Table 2 shows that the studies from the supported centres have covered a wide range of the ICPD PoA. HRP External Evaluation 2008−2012 Chapter 5 Page | 168 Table 1 Completed studies by 12 centres receiving long-term institutional development and resource maintenance, capital and small supplies grants in the African and Eastern Mediterranean Regions in 2009 and 2010 Thematic area Completed studies Funding National International WHO Maternal health 5 3 1 1 Family planning 3 3 Unsafe abortion 1 1 HIV/AIDS 3 2 1 Other 7 4 1 2 Total 19 10 4 5 Table 2 Research studies conducted by centres receiving support from HRP in the Region of the Americas in 2009 and 2010 Thematic area Year 2009 2010 Number % Number % Adolescent reproductive health 24 8 23 12 Family planning 42 15 42 22 Health systems 17 6 14 7 HIV 25 9 34 18 Infertility 20 7 4 2 Maternal and neonatal health 40 14 34 18 Reproductive biology 60 21 14 7 Reproductive cancers 16 6 6 3 RTIs/STIs 10 3 5 2 Unsafe abortion 2 1 2 1 Gender, violence, harmful practices 27 10 7 4 Menopause, other reproductive health issues — — 8 4 Total 283 100 193 100 RTI, reproducctive tract infection; STI, sexually transmitted infection. The major output attributable to HRP in the supported centres was increase in high-quality research proposals and output of papers published in peer-reviewed journals (83%). At the Instituto de Biología y Medicina Experimental (IByME) [Institute of Experimental Biology and Medicine] in Argentina, HRP support more than doubled the number of qualified researchers on the centre’s staff and publications increased five-fold. Between 2008 and 2012, the centre in Viet Nam carried out 51 studies that were published in 28 peer-reviewed journals. The staff also participated in 45 international research projects. This has promoted evidence-based treatment and programming of health care in low- and middle-income countries2. HRP 2 In the 15 institutions assessed, 922 people were trained through short-term courses, while 7 had long-term postgraduate training. A recent study by interns Oscar Zazueta-Fierro and Alfredo Fort (personal communication, 14 HRP External Evaluation 2008−2012 Chapter 5 Page | 169 activities have also promoted universal coverage for SRH services, for example by reducing antenatal care visits, while at the same time improving the quality of care through the adoption of focused antenatal care after research by HRP-supported centres demonstrated its effectiveness. However, institutions complained about the lengthy process of requesting support and evaluation, approval and release of funds. 3.3 Development of regional networks Creation of networks has promoted development of global public goods. Although, understandably, emphasis has been on support for low- and middle-income countries, collaboration with high-income countries has led to development of global goods such as methods for medical abortion. This collaboration has included multicentre research on family planning, with a special emphasis on methods of contraception for men. Other projects include implantable contraceptives, with participation from Brazil, Chile, the Dominican Republic, Hungary, Thailand and Turkey. RHR continues to strengthen the global collaborative research and implementation efforts of institutions and individuals working to reduce maternal and newborn deaths. In maternal and newborn health, a multinational collaborative study of gene– environment interactions in spontaneous preterm birth, in which the United Kingdom of Great Britain and Northern Ireland and Denmark are participating, and the study “Screening for pre- eclampsia: evaluation of the predictive ability of angiogenic factors for pre-eclampsia” (5), with participation of Switzerland and Italy are two examples of this collaborative research. Thirty- five per cent of the centres reported collaboration with regional networks (3 in Latin America, 2 in Asia and 3 in Africa). These all feel the networks are very useful for exchanging views and learning. They also all feel that the networks need more funding to meet the needs of their regions. The African Network for Research and Training in Sexual and Reproductive Health and HIV (ReproNet-Africa) currently has 11 Member States (Cameroon, Egypt, Kenya, Malawi, Mali, Nigeria, South Africa, the United Republic of Tanzania, Uganda, Zambia and Zimbabwe) sharing information and influencing policy and practice. Eighty-seven per cent of the centres in low- and middle-income countries are contributing to formulation and implementation of regional research priorities, to varying extents. The Centro de Estudios de Población (CENEP) [Centre for Population Studies] in Argentina coordinated research on “Reality and beliefs in the sexual and reproductive decision making process: men's perceptions and behaviour” carried out by researchers in Argentina, the Plurinational State of Bolivia, Cuba and Peru. They together developed the instrument for a survey that could yield comparative results and be culturally sensitive to local conditions. In the process, the centre also strengthened its capacities. In Argentina, the Centro Rosarino de Estudios Perinatales (CREP) [Centre for Perinatal Studies] is not only a resource for the Region of the Americas, where it has been engaged in multicentre studies (including in coordination roles) using randomized controlled trials, and systematic reviews in perinatal health, but it also trains researchers from other regions. For example, it has trained researchers from Thailand, who are in turn cascading the training in data management in the South-East Asia and Western Pacific Regions. At the request of the Government, the centre in Peru has also supported studies there, just as the centre in Burkina Faso had a memorandum of understanding with UNFPA to conduct emergency obstetric and neonatal care (EmONC) assessments in other countries in West Africa, after successfully doing the same in Burkina Faso. The reproductive health centres that participated in the six countries September 2012), revealed that between 2008 and 2011 there were 65 who benefited from long-term training (50 in the Region of the Americas, 6 in the African Region, 4 in the South-East Asia Region and 5 in the Western Pacific Region). HRP External Evaluation 2008−2012 Chapter 5 Page | 170 of Madagascar, Malawi, Nigeria, Uganda, the United Republic of Tanzania and Zambia conducted a study on the acceptability and feasibility of the technique of visual inspection with acetic acid (VIA) for prevention of cervical cancer (6), which has become the normal screening procedure for cervical cancer in the region and in many other countries, such as Viet Nam. ReproNet-Africa participated in development of the regional agenda for accelerating universal access to proven effective reproductive health services in the African Region, using the most effective and appropriate delivery system. Seventy per cent of the centres had also contributed to formulation and implementation of national research priorities, to varying extents, to promote collaboration between the research centres and the ministries of health in their countries. The centres in Argentina, Burkina Faso, Mongolia and others in South Africa are subcontracted by governments to conduct prioritized studies. Works on partner violence, fetal loss, reproductive health care and family planning and research on the demography and reproductive health surveys in Paraguay were all included in the Paraguay National Agenda of Priorities in Health Research 2008–2013. In Zimbabwe, studies from the centre there have led to development of the national Adolescent Reproductive Health Strategy. In Ethiopia, the centre’s work led to the introduction of a maternal death audit project and national cervical cancer screening programme. In Eastern Europe and Central Asian republics, a study exploring effective strategies to reduce maternal mortality was conducted through the collaboration of six countries. The findings revealed strategies that have worked in reducing maternal deaths as: (1) introducing innovative financing measures; (2) strengthening preservice education and in-service training for health-care providers; (3) enhancing obstetric care including infrastructure, equipment and quality of services; and (4) investments in the broader determinants of maternal mortality, particularly family planning, education and women’s empowerment. Studies from collaborative centres gave the results of the “Active management of third stage of labour trial”, published in The Lancet in 2012 (7), and concluded that controlled cord traction can be omitted with little increase in the risk of postpartum haemorrhage in settings where skilled birth attendants are not available. In these settings, injection of intramuscular oxytocin after birth will contribute greatly to the reduction of the primary cause of maternal death – haemorrhage. A number of multicountry studies on misoprostol were also conducted. As a set, these studies, from diverse cultural, legal and services contexts, provide evidence of high uptake of contraception following abortion and that there was no difference in the level of uptake or continuation of use among women accepting medical abortion, as compared with those having surgical abortion. These studies also show that uptake of medical abortion is significantly increasing, and is acceptable, with potential to significantly reduce maternal mortality if fully utilized. A study that showed that midlevel health-care providers can provide medical abortion as safely and effectively as physicians was published in The Lancet in 2011 (8). Where there are not enough doctors, as in most low- and middle-income countries, lack of task-shifting of abortion services is a barrier to provision of abortion services as permitted by national laws. This study has shown, unequivocally, that appropriately trained midlevel providers are able to do both surgical and medical abortions as competently as doctors in relatively low-resource settings, where the need is greatest, and this has the potential to reduce abortion mortality and accelerate the achievement of MDG 5. The centre in Viet Nam (Hung Vuong Hospital – HVH) is currently collaborating with Gynuity Health Projects in three other multicentre trials on medical abortion, to investigate the potential effectiveness of misoprostol alone for different indications in medical abortion. Eighty-three per cent of the supported centres collaborated with other institutions: of these, 61% collaborated with other local institutions, thereby cascading the knowledge and skills HRP External Evaluation 2008−2012 Chapter 5 Page | 171 benefit from HRP support, while 39% collaborated with regional institutions as well as global/international organizations. This all promotes the HRP objectives to provide evidence- based practice in the delivery and provision of clinical reproductive health services, thereby accelerating the achievement of MDGs. Through training and provision of logistics support by HRP, a culture of research was set in many low- and middle-income countries and the output of quality research that informs policy and programmes has increased. The research capacity-building objectives are therefore either achieved or likely to be achieved soon. 3.4 Outputs of HRP-strengthened research centres are being used to improve health and save lives There are many examples of research results that have led to improvement in women’s lives. For example, a study that concluded that controlled cord traction can be omitted with little increase in the risk of postpartum haemorrhage in settings where skilled birth attendants are not available (7), has potential to save many lives in low- and middle-income countries where postpartum haemorrhage remains the top cause of maternal death. A study by a collaborating centre on “maternal haemoglobin and pregnancy and fetal outcome in high altitude in Peru”, revealed that there was a relationship between very low and very high haemoglobin levels and stillbirths (9). Additionally, given results indicating an artificial increase of adverse outcomes due to changes in anaemia definitions because of altitude, researchers have put forward the recommendation not to use an adjustment factor to correct haemoglobin levels in high altitudes, as this would erroneously lower haemoglobin levels and lead to unnecessary iron supplementation that could result in poor fetal outcomes. In Mongolia studies by the HRP-supported centre led to introduction of modern contraceptive methods for all age groups and the legalization of first-trimester abortion, which together are accelerating the reduction of maternal and child mortality rates. In Myanmar, following the centre’s RCS grant, guidelines for midwives were developed, entitled “Strengthening of quality antenatal care focusing on pre-eclampsia”, which is now being used by the Ministry of Health in Myanmar, leading to improve obstetric outcomes. The WHO multicountry survey on maternal and newborn health, with a focus on the management of severe complications in pregnancy and childbirth in 29 countries (10), has the potential to reduce the rates of case- fatality from obstetric complications and save many lives. Through network building, HRP has promoted interregional collaboration in promotion of women’s and children’s health. Examples of this include the centre in Argentina, which was requested by the Government of Peru to support studies in their country. The centre in Burkina Faso has a memorandum of understanding with UNFPA to conduct EmONC assessments in other countries in West Africa, after successfully doing the same in Burkina Faso. The CENEP in Argentina has also been part of the development of the National Research, Technology and Innovation Policy for Health, approved by presidential decree. This policy recommends strategic guidelines aimed at strengthening structures for health research and consolidation of a national researchers’ career and human resource training, and promotes actions to overcome asymmetries and gaps in health research. 3.5 Sustainability of HRP’s work From the start, the supported institutions are encouraged to develop a sustainability plan to allow them to continue thriving beyond the period of support. A number of institutions have since been weaned off HRP support, in many countries including Argentina, Brazil, Kenya, Senegal, Tunisia and Zimbabwe, but have continued to turn out large quantities of good- HRP External Evaluation 2008−2012 Chapter 5 Page | 172 quality research results. All the eight institutions that had been weaned off support during the period of this evaluation have obtained LID grants from other organizations, such as the United Nations Children’s Fund (UNICEF), UNFPA, UNDP, the Pan American Health Organization, their ministry of health, Emory University, the Gates Institute for Population and Reproductive Health, the World Bank, the United States Agency for International Development/Family Health International and the National Institutes of Health (NIH)/Partnership in Innovative Medical Education Kenya. In Sri Lanka, the World Bank has, since then, entered into a national- level institution-strengthening programme to encourage a research programme, as well as a health sector development project in the Ministry of Health. In 2011, 10 institutions received RMC grants to ensure their continued development after the LID grants. Eighty-nine per cent of the centres collaborated with other institutions: 63% collaborated with other local institutions, thereby cascading the knowledge and skills benefit from HRP support, while 37% collaborated with regional institutions and with global/international institutions. After the LID grant, the institution in Vietnam (HVH) was able to establish networking with national and international centres and to participate in many multicentre and international trials. The centre is currently collaborating with Gynuity Health Projects in three other multicentre trials on medical abortion, to investigate the potential effectiveness of misoprostol alone for different indications in medical abortion. In some countries, this collaboration has enhanced the capacity to attract funding from other institutions. For example CREP in Argentina has been able to attract diverse funding towards its activities. Its current portfolio includes funding from UNICEF-Argentina, Nestlé Foundation, the Argentinian Ministry of Health, the London School of Hygiene and Tropical Medicine, Brussels University, University of Oslo, the Pan American Health and Education Foundation, Gynuity, the National Perinatal Epidemiology Unit of the University of Oxford, the European Commission, the Ottawa Hospital Research Institute, and the NIH (United States of America). The centre in Tunis had LID grants in 1978–1992, and 1997–2002, followed by a SSG. In 2010, the centre had 134 academic activities, 123 at national level and 11 at the international level. Other centres in Africa are also now attracting research grants and participating in multicentre studies funded by WHO or other partners. However, the ending of a LID grant has led to the loss of salaries for some research staff in some centres, which has resulted in their leaving, with consequent weakening of the centres. This reduced the capacity of centres to train others. In Viet Nam for example, of the seven researchers trained, four have retired without replacement as the long-term training dried up. Other services that have suffered are library services and journal acquisition, which have become depleted (e.g. Sri Lanka). At IByME in Argentina, when the institution stopped getting the LID grant, the number of fellows working in the group was significantly reduced because the resource maintenance grant (RMG) does not allow the payment of salaries to personnel, and the funds for fellowships provided by the national government have been significantly reduced during recent years. ReproNet-Africa feels that the sustainability of the network remains uncertain, despite several efforts to seek funding from prospective donors. This has led to ReproNet’s inability to explore its full potential, as a result of inadequate financial support. 3.6 HRP work through research capacity strengthening and network building has impacted on women’s reproductive health and children’s health globally HRP’s work has resulted in many changes in policy and clinical practice that have led to improvements in women’s reproductive health and children’s health globally. While many centres are not monitoring the impact of their work on policy and practice, most centres’ work is influencing national SRH policies that have improved women’s health, or processes that are HRP External Evaluation 2008−2012 Chapter 5 Page | 173 known to improve women’s and children’s health. Among recorded policy changes and practice are the use of emergency contraception and provision of safe medical abortion care in many countries worldwide; adoption of the use of magnesium sulfate in the management of pregnancy-induced hypertension; use of misoprostol in the management of postpartum haemorrhage and incomplete abortion; and replacement of dilatation and evacuation with manual vacuum aspiration (11). Adoption of the one-stop antenatal screening for syphilis in Mongolia improved newborn health. In Zimbabwe, the centre’s work led to the development of national clinical guidelines for reproductive health. In Burkina Faso, the centre’s study on contraceptive products has enabled the National Direction de la Santé de la Famille (DSF) to review the follow-up of availability of contraceptive products. In Viet Nam, HRP collaboration promoted evidence-based clinical practice countrywide. For example, before the collaboration, doctors delivered breech presentations at term vaginally; nowadays, most breech presentations are delivered by caesarean section, thereby dramatically reducing neonatal mortality from breech delivery. All clinicians are being trained to perform external cephalic version for breech presentation at 36 weeks, to decrease the rate of caesarean section. Introduction of prophylactic antibiotics before surgery, as a national policy in Viet Nam, reduced the cost for treatment of hospital-acquired infections over the years from 7% to 2% of the hospital’s (HVG’s) overall budget. Together, these have led to significant reductions in maternal mortality and morbidity and significant cost savings in the delivery of health care. As a result of regional research collaborations through the establishment of networks in the regions, many guidelines and standards of care that are universally accepted have been developed based on regional research outcomes. Protocols on the management of major causes of maternal and neonatal morbidity and mortality have been produced to reduce their rates and thereby accelerate the achievement of MDGs. Although studies have not been conducted to assess the impact of centres on health-related MDGs in their countries, there is evidence that work by supported centres has contributed to the achievement of MDGs in their countries. For example, studies in Peru revealed that a haemoglobin level above 14.5 g/dl during pregnancy was associated with small-for-date births (9), and therefore supplementation with iron for women in high-altitude areas where there was a low rate of anaemia was discouraged, thereby contributing to a reduction of neonatal mortality. In Côte d’Ivoire, the results of the EmONC survey made government introduce the delivery of free care for children and pregnant women, thereby increasing coverage, which will accelerate the achievement of MDGs 4 and 5 in Côte d'Ivoire. In Viet Nam, HVH collaborated with WHO task forces to investigate the effectiveness of several interventions, in an effort to minimize maternal morbidities and mortalities through calcium supplementation or supplementation with vitamins C and E for prevention of pre-eclampsia, or misoprostol for prevention and treatment of postpartum haemorrhage. In Peru, a study demonstrated that burning biofuel led to reductions in birth weight over and above the effect produced by living in a hypoxic environment. Further studies led to the introduction of biofuel kitchens in hypoxic high-altitude areas, which reduces the incidence of low birth weight (12). It is estimated that the use of 500 000 such improved kitchens may reduce the rates of low birth weight. The centres have also contributed to the promotion of the ICPD PoA in their countries. For example, in Ethiopia some of the studies that were conducted by the centre were on family planning and abortion, which is part of the ICPD/PoA. In Kenya, guidelines for adoption of family planning and promotion of contraception among women who are HIV positive have promoted the ICPD agenda. In the Plurinational State of Bolivia, the centre has participated in activities promoting Cairo+15, and now Cairo+203. As result of institutional research agenda, 3 ICPD met in 1995 in Cairo and produced a 10-year Programme of Action to 2005 (3). This was then modified for 15 years to 2010 (Cairo+15) and then for 20 years to 2015 (Cairo+20). HRP External Evaluation 2008−2012 Chapter 5 Page | 174 new topics have been introduced in population studies, such as interculturality, violence against women, teenage pregnancy and motherhood. Implementation research is increasing, but rather slowly, probably because it is expensive. The few studies that have been conducted include a study on the “Coverage of antenatal syphilis screening and predictors for not being screened in Ulaanbaatar, Mongolia” (13), which found that the coverage of antenatal syphilis screening was still low in the country, with poor contact tracing. The results were used to develop an operational research proposal on one-stop service on antenatal syphilis screening. The results of the one-stop service on antenatal syphilis screening project findings (14) influenced the Government of Mongolia to reconsider the national policy revision regarding antenatal syphilis screening. On discovering that the country could not afford to scale up the Papanicolaou smear test for cervical cancer screening, the United Republic of Tanzania participated in a six-country (ReproNet) VIA study (6), which found that VIA was acceptable and feasible. VIA has since been implemented in these countries, leading to the scaling up of screening services for cervical cancer, the commonest cancer in women in low- and middle-income countries. 3.7 The extent to which HRP contribute to global goods An example of global goods from HRP-supported research is the non-scalpel vasectomy developed by the Sichuan Family Planning Research Institute in Chengdu, China. This method is now practised worldwide and has made vasectomy more acceptable and accessible to many individuals. Research on the development of the award-winning Odon device, which can potentially be safely used by nurses in rural health centres without risk of maternal and newborn trauma, is another global good that could provide a low-cost simplified way to shorten the second stage of labour without recourse to caesarean section. Clinical studies on the Odon device were being coordinated by a centre in Argentina that had been a recipient of an HRP RCS grant. Among the global goods whose popularization will lead to the saving of many lives, especially in low- and middle-income countries where there is a shortage of the health-care workforce, is misoprostol, which is now registered in many countries. 3.8 The extent to which capacity-strengthening and network-building outputs have reflected the mandate of HRP Through training and provision of logistics support by HRP, a culture of research has been established in many low- and middle-income countries and the output of quality research that informs policy and programmes has increased. The research-capacity-building objectives are either achieved or likely to be achieved with time. Among the outputs of the investment in RCS is the increase in research staff in research institutions, leading to increased output of studies that have been accepted for publication by peer-reviewed journals. 3.9 The comparative advantage of HRP for research-capacity strengthening and network building and as global leader for research in human reproduction WHO remains the organization that countries look up to for guidance on health promotion, and countries are more likely to adopt HRP guidelines, which are informed by HRP-supported studies globally, than those produced by other partners. There are a number of other players in the area of research capacity-development in reproductive health in low- and middle- income countries, including the Buffet Foundation and the Bill and Melinda Gates Foundation, and the Centers for Disease Control, but they are mostly either supporting or complementing HRP External Evaluation 2008−2012 Chapter 5 Page | 175 HRP and not competing with it, as HRP is seen to be better placed to influence policy and programmes. Studies funded by HRP, by being focused on country needs, have led to the development of treatment guidelines and global standards that are used in SRH programmes worldwide, focusing on the five key causes of maternal mortality, notably postpartum haemorrhage, infection, eclampsia, obstructed labour, and complications resulting from unsafe abortion. Among these are the award-winning Medical eligibility criteria for contraceptive use (15), Safe abortion: technical and policy guidance for health systems (16) and guidelines on postpartum haemorrhage (17). There are also many treatment guidelines: such as Safe abortion: technical and policy guidance for health systems (18), Recommendations for induction of labour (19) and Recommendations for prevention and treatment of pre-eclampsia and eclampsia (20). Seventy-five per cent of the institutions that responded to this question felt that this advantage was more specific to SRH research issues; 45% also felt that this was related to the high-quality training provided through sponsorship to good institutions and the provision of experts for local training. One third also reported that HRP promoted studies that were more specific to national needs. 3.10 The extent to which HRP work has been relevant to programme countries, particularly to low income countries HRP outputs have been accessible to all and used to improve lives. A shortage of human resources is critical in most low-income countries. Studies that provide evidence that service coverage can be safely scaled up with task-shifting or sharing have been welcomed. Malawi increased the prevalence of contraception from 28% to 42% when depot medroxyprogesterone was made available to women through the community-based distribution service-delivery system. Likewise, the provision of first-trimester abortion by non- physicians, as in South Africa, has led to a significant fall in abortion-related deaths. HRP’s RCS initiatives are also influencing the development of women’s and children’s programmes in many countries. For example, in Burkina Faso, Nigeria and Zimbabwe, among others, research staff from the collaborating centres sit on national SRH research policy planning bodies, thereby influencing the development of reproductive health programmes in their countries, and in some cases in their region. In Paraguay, studies on partner violence, fetal loss, reproductive care, and family planning conducted by a LID grant recipient appear on the Paraguay National Agenda of Priorities in Health Research 2008–2013. Among recorded policy changes and practice are the adoption of the one-stop antenatal screening for syphilis in Mongolia (12) instead of the conventional screening, which led to increased uptake of the service; and the use of haemoglobin colour scale in the estimation of anaemia in pregnant women in Myanmar and other countries. In Zimbabwe, the centre’s work led to the development of national clinical guidelines for reproductive health. In Malawi, the centre’s Cervical Cancer Project report informed national SRH policy, while in Zimbabwe, studies of the centre have led to development of the country’s adolescent reproductive health strategy. In Burkina Faso, the centre’s study on contraceptive products has enable DSF to review the follow-up of contraceptives product availability, thereby promoting contraceptive security. In Peru, studies led to the introduction of biofuel kitchens in hypoxic high-altitude areas that have reduced the incidence of low birth weight. HRP External Evaluation 2008−2012 Chapter 5 Page | 176 3.11 The extent to which HRP’s work in research-capacity strengthening is contributing to the achievement of MDGs, ICPD agenda, poverty reduction and women’s health in general HRP has funded the training of large number of scientists in research methodology and postgraduate education for senior researchers, which has influenced policy and programmes in low- and middle-income countries and promoted evidence-based treatment and programming of health care in low- and middle-income countries. This is promoting universal coverage of SRH services, while also improving the quality of care, e.g. through the adoption of focused antenatal care. HRP has been very effective in developing the research capacity of collaborating institutions. The major outputs attributable to HRP in the collaborating centres include an increase in high- quality research proposals and paper output, and the development, or revision, of national guidelines. This is contributing to achievement of national MDGs; countries have also benefited from HRP collaboration through the promotion of the ICPD/PoA. For example, the preservice training curriculum in the Health Sciences University of Mongolia was updated to reflect new national standards adapted from those of HRP. 3.12 Programme strengths and weaknesses Fifty-three per cent of the centres that responded to the questionnaire indicated that HRP fully met their expectations in RCS. Shortfalls identified were inadequate funding to meet their needs, including the training of adequate numbers of researchers and support to attend international meetings and share the results of studies. Another area that requires strengthening is the translation of research into policy. Many researchers still see the end results of their work as acceptance of papers by peer-reviewed journals. There is little follow-up on the impact of their work on policy and practice within their countries or region. Thirty-two per cent of the centres reported that HRP visits were not adequate to provide the support that the centres needed. Some reported that in the 10 years of the LID grant, they could recall only one HRP visit. 4 Conclusions Testimonials from institutions consider the HRP support as the only real possibility to train with first-line researchers in the area of reproduction. Workshops and seminars were found to be an enriching experience that creates a scientific frame of mind. There were few financial opportunities to travel to training centres for most researchers in low- and middle-income countries, other than those provided by HRP. Many young researchers have gone on to obtain masters degrees or doctorate degrees and returned to their jobs with increased professional advancement and contributing significantly to the research output of their centres, thereby creating a critical mass of researchers who have a significant role in formulating national and global SRH policies and care. The result is a greatly increased output of publications in peer- reviewed journals that have influence on policy and practice the world over, for the improvement of women’s and children’s health. In the period evaluated, HRP’s RCS efforts continue to be among its most successful initiatives. The major facilitating factors for the achievement of objectives include the use of RAPs to ensure support to the prioritized needs of countries. The use of RAPs has ensured that the activities of the programme are relevant to the regions and that the activities and outputs of the programme are consistent with the intended impacts and effects. Development of networks to promote the development of global goods, and the promotion of sharing of HRP External Evaluation 2008−2012 Chapter 5 Page | 177 information and skills in research have also been effective in strengthening research capacity and accelerating improvement in women’s health. There are also partnerships that HRP has developed with other development partners, within and outside the UN system, that are providing extra support to the research institutions. However, gaps still remain in the needs of the countries, as a result of inadequate funding to satisfy the needs of the centres. In the face of reducing funding to HRP, it remains to be seen whether reductions in the amounts of RCS grants and the lengths of study will result in a slower pace of RCS by HRP. 5 Lessons learnt The chances that research will influence policy are increased when the programme managers are involved in identification of the study topics or are involved in the conduct of the study, as occurs where centre staff are on the national committees that prioritize research topics, and when the researcher has a one-to-one dissemination meeting with the programme manager. Where health care is decentralized, district health-care managers who attend dissemination meetings are more likely to use study results. Involvement of senior public-sector clinicians in a study is more likely to lead to change in practice. Some institutions are still not able to stand on their own after the 10-year LID grant. Many centres in low-income countries will still not have an adequate research staff at the end of the 10-year support for institutional development. This is partly because of mobility of staff and partly due to countries’ dependence on HRP support. In Viet Nam, when four of the seven long-term training researchers retired, gaps were created in the leadership of the centre. The ending of LID grants in some institutions has led loss of salaries for some research staff, which resulted in their leaving, with consequent weakening of the centres. This has reduced the capacity of centres to train others. These are, however, exceptions rather than the rule, as many institutions got LIG grants from other institutions and RMGs from HRP. Networks are appreciated in the regions. Fifty-eight per cent of the centres that responded to the questionnaire were collaborating with other local institutions, thereby cascading knowledge and skills benefits from HRP support. Six others were collaborating with regional and global institutions. Research institutions that are not utilizing the networks established in their regions lack understanding of their value, or lack financial support to participate in the networks. A frequently mentioned support gap is inadequate support to attend regional and international scientific meetings. Those who know these networks, however, feel that the networks are very useful in exchanging views and learning. However, respondents from all centres all feel that these networks need more funding to meet the needs of their regions. 6 Recommendations 1. To promote sustainability, LID grant support should have a clear exit strategy, which should be continuously monitored, including lobbying with national governments to sustain the centres when the HRP support comes to an end. In Benin and Cameroon, the research centres collapsed when the leading researchers left, because the objective of HRP to build up national self-reliance in research, with a view to developing a critical mass of scientists, had not been achieved at the end of the LID support. 2. Likewise, long-term scholarships should continue until centres have a full complement of staff, and provide more support to countries to carry out their research priorities, even if it means having fewer active collaborating centres at a time. Achievement of institutional development, including an adequate complement of staff, should be structured in the LID grant support and monitored throughout the period of support. HRP External Evaluation 2008−2012 Chapter 5 Page | 178 Governments should be engaged in the institutional development, to ensure that their inputs, including the transfer of staff from the centres, should be in line with the objectives of the HRP support. 3. In addition, it is recommended that LID grant implementation should include lobbying with national governments to sustain the centres as the aid comes to an end. 4. HRP could better serve countries by expanding and strengthening the regional networks to benefit countries in the regions with resources available in other countries in the region, such as from Argentina in Latin America. 5. Research grants should include a budget for dissemination and follow-up on the utilization of the research results. Appropriate dissemination processes targeting policy-makers and programme managers should be clearly outlined. University departments of obstetrics and gynaecology are critical in changing clinical practice and should be a target of meetings for dissemination of research results that are not from the university itself. 6. The objectives of studies should include deliberate steps to influence policy and programmes, with clear methodologies on how that is to be achieved. Justification of funded studies should be clear on how they would influence policy and practice, and how the researchers could influence policy-makers and programme managers. Targets for change should include health-care worker training institutions, by influencing the content of curricula. 7. HRP should promote studies that evaluate the impact of support provided on outcomes; prioritizing health system strengthening. A lot of information is now available but it needs to be put it into use in programmes. 8. More studies designed to promote MDG 5B should be encouraged in low- and middle- income countries, to accelerate improvements in women’s reproductive health and neonatal health. 9. Support to regional networks should include funding that ensures that every country in the supported region is seen to be participating in the network. 10. HRP should develop a long-term strategy for RCS within HRP that includes maintaining healthy levels of funding to sustain initial support for individuals and institutions. This would include proactive identification of potential donors (including within countries and regions) and aggressive proposal writing for calls for proposals, funding of regional initiatives (e.g. adolescent pregnancy) and dissemination of research results. 11. The centres feel that HRP can best serve them by: 11.1 expanding the regional networks and strengthening them with resources, to benefit other countries, with resources available in other countries in the region, such as from Argentina in Latin America; 11.2 promoting exchange programmes and increasing staff development, with support for staff exchange programmes, and involvement of centres in HRP multicentre trials; 11.3 providing more long-term scholarships to ensure centres have a full complement of staff; 11.4 providing more opportunities for dissemination of results in international conferences through travel grants and encouraging local scientific meetings in research dissemination; HRP External Evaluation 2008−2012 Chapter 5 Page | 179 11.5 providing postdoctoral fellowships that allow those human resources that are already fully trained to continue contributing to scientific research in reproduction; 11.6 increasing financial and technical support for developing health research systems, supporting institutional linkages in areas of research, and strengthening human resource capacity; 11.7 revisiting the processing of applications for LID grants; it was observed that it is very lengthy, and the review committee changes its observation in each review! As a result, the approval process takes more than 2 years, by which time the budget is outdated. HRP External Evaluation 2008−2012 Chapter 5 Page | 180 References 1. World Health Organization, Department of Reproductive Health and Research. Reproductive health strategy: to accelerate progress towards the attainment of international development goals and targets. Geneva, World Health Organization, 2004 (WHO/RHR/04.8) (http://whqlibdoc.who.int/hq/2004/WHO_RHR_04.8.pdf, accessed 19 December 2012). 2. A strategy for research capacity strengthening in developing countries. Geneva, World Health Organization, undated (http://www.who.int/reproductivehealth/topics/countries/lid.pdf, accessed 14 January 2013). 3. United Nations Population Fund. Programme of Action. Adopted at the International Conference on Population and Development, Cairo, 5–13 September 1994.) New York, United Nations Population Fund, 1995 (A/CONF.171/13/Rev.1 – Report of the International Conference on Population and Development. (http://www.unfpa.org/public/publications/pid/1973?, accessed 15 January 2013). 4. United Nations Secretary-General. Global strategy for women’s and children’s health. New York, United Nations, 2010 (http://www.who.int/pmnch/topics/maternal/201009_globalstrategy_wch/en/index.h tml, accessed 19 December 2012). 5. von Dadelszen P et al. Prediction of adverse maternal outcomes in pre-eclampsia: development and validation of the full PIERS model. The Lancet, 2011, 377:219–227. 6. Prevention of cervical cancer through screening using visual inspection with acetic acid (VIA) and treatment with cryotherapy. A demonstration project in six African countries: Malawi, Madagascar, Nigeria, Uganda, the United Republic of Tanzania, and Zambia. Geneva, World Health Organization, 2012. 7. Gülmezoglu AM et al. Active management of third stage of labour with and without controlled cord traction: a randomised, controlled, non-inferiority trial. The Lancet, 2012, 379:1721–1727. 8. Warriner IK et al. Can midlevel health-care providers administer early medical abortion as safely and effectively as doctors? A randomised controlled equivalence trial in Nepal. The Lancet, 2011, 377:1155–1161. 9. Gonzales GF et al. Association of haemoglobin values at booking with adverse maternal outcomes among Peruvian populations living at different altitudes. International Journal of Gynecology and Obstetrics, 2012, 117:134–139. 10. WHO multicountry survey of maternal and newborn health 2010–2012. Geneva, World Health Organization. 11. Kulier R et al. Medical methods for first trimester abortion. Cochrane Database of Systematic Reviews, 2011, (11):CD002855. 12. Yucra S et al. Association between biofuel exposure and adverse birth outcomes at high altitudes in Peru: a matched case-control study. International Journal of Occupational and Environmental Health, 2011, 17:307–313. 13. Munkhuu B et al. Coverage of antenatal syphilis screening and predictors for not being screened in Ulaanbaatar. Sexually Transmitted Diseases, 2006, 33:284–288. HRP External Evaluation 2008−2012 Chapter 5 Page | 181 14. Munkhuu B et al. One-stop service for antenatal syphilis screening and prevention of congenital syphilis in Ulaanbaatar, Mongolia: a cluster randomized trial. Sexually Transmitted Diseases, 2009, 36:714–720. 15. Department of Reproductive Health Research, World Health Organization. Medical eligibility criteria for contraceptive use, 4th ed. Geneva, World Health Organization, 2009. 16. World Health Organization, Department of Reproductive Health and Research. Safe abortion: technical and policy guidance for health systems, 2nd ed. Geneva, World Health Organization, 2012. 17. WHO guidelines for the management of postpartum haemorrhage and retained placenta. Geneva, World Health Organization, 2009. 18. World Health Organization, Department of Reproductive Health and Research. Safe abortion: technical and policy guidance for health systems, 2nd ed. Geneva, World Health Organization, 2012. 19. World Health Organization, Department of Reproductive Health and Research. WHO recommendations for induction of labour. Geneva, World Health Organization, 2011. 20. WHO recommendations for prevention and treatment of pre-eclampsia and eclampsia. Geneva, World Health Organization, 2011. HRP External Evaluation 2008−2012 Chapter 5 Page | 182 Annex 1: Centre recipients of long-term institutional development grants from HRP that were sent a questionnaire by the external evaluator Number Centre Responded African Region 1 Burkina Faso Institut de Recherche en Sciences de la Sante (IRSS) Yes 2 Cellule de Recherche en Santé de la Reproduction – Côte d'Ivoire (CRESAR-CI) Yes 3 Democratic Republic of the Congo Cliniques Universitaires de Kinshasa Yes 4 Department of Obstetrics and Gynaecology – Ethiopia Addis Ababa University Yes 5 Guinea Cellule de Recherche en Santé de la Reproduction en Guinée (CERREGUI) No 6 University of Nairobi Department of Obstetrics and Gynaecology – Kenya (Zahida Qureshi) Yes 7 Centre for Reproductive Health, Malawi University of Malawi Yes 8 Nigeria Sagamu Centre for Research in RH (CRRH) Yes 9 South Africa ECRU Frere Maternity Hospital No 10 United Republic of Tanzania Kilimanjaro Christian Medical Centre (KCMC) Yes 11 Zambia ReproNet Yes 12 Department of Obstetrics and Gynaecology, College of Health Sciences, University of Zimbabwe Yes Region of the Americas 13 Centro de Estudios de Población (CENEP) – Argentina Yes 14 Instituto de Medicina y Biología Experimental (IBYME) – Argentina Yes 15 Centro Paraguayo de Estudios de Población (CEPEP) – Paraguay Yes 16 Instituto de Investigaciones de la Altura, Universidad Peruana Cayetano Heredia – Peru Yes 17 Postgrado en Ciencias del Desarrollo – Universidad Mayor de San Andrés (CIDES-UMSA) – Plurinational State of Bolivia Yes South-East Asia and Western Pacific Regions 19 Epidemiology and Research unit, Ministry of Health, Bhutan Yes 20 National Institute of Public Health, Phnom Penh, Cambodia Yes 18 All India Institute of Medical Sciences, India Yes 22 National Centre for Maternal and Child Health (formerly State Research Ct for MCH) – Mongolia Yes 21 Department of Medical Research – Upper Myanmar Yes 23 National Coordinating Committee on Reproductive Health Research of Sri Lanka Yes 24 Hung Vuong Hospital, Viet Nam Yes Eastern Mediterranean Region 25 Afghanistan National Public Health Institute – Ministry of Public Health Yes Total (OF 25 e-mails sent) 23 (92%) Page | 183 Chapter 6 Strengthening implementation research Fernando Althabe Director, Department of Mother and Child Research Institute for Clinical Effectiveness and Health Policy in Buenos Aires, Argentina Daniela S Colaci Researcher, Department of Mother and Child Research Institute for Clinical Effectiveness and Health Policy in Buenos Aires, Argentina Page | 184 Page | 185 Contents Abstract . 187 Introduction . 187 Methods . 187 Key findings, comments and conclusions . 187 Recommendations . 188 1 Introduction . 189 1.1 What is implementation research? . 189 2 Rationale. 189 2.1 The problem: underuse of life-saving interventions in low- and middle-income countries . 189 2.2 Scaling up life-saving interventions . 190 2.3 Why has this topic has been included in this evaluation? . 190 3 Methods . 191 4 Key findings. 192 4.1 Research governance milestones at HRP related to implementation research . 192 4.2 Implementation research within the HRP strategic plan for 2010–2015 . 193 5 Special initiatives and projects . 193 5.1 The Implementation Research Platform . 193 5.2 The GREAT project (guideline development, research priorities, evidence synthesis, applicability of evidence, transfer of knowledge). 194 5.2 Global survey on maternal and perinatal health . 195 5.3 The WHO Strategic Approach to strengthening sexual and reproductive health policies and programmes . 195 6 Research projects . 196 7 HRP peer-reviewed publications 2008–2012 . 200 8 Conclusions . 200 8.1 Positive findings . 200 8.2 Pending issues . 201 9 Recommendations . 202 References . 204 Annex 1: Implementation research projects undertaken by HRP/RHR between 2008 and 2012 . 206 Annex 2: Publications in peer-reviewed journals related to implementation research projects . 217 Page | 186 Abbreviations ANC antenatal care AMTSL Active Management of the Third Stage of Labour (trial) CHW community health worker DFID UK Department for International Development EBM evidence-based medicine ERC Ethics Review Committee GP general practiitoner GRADE grades of recommendation, assessment development and evaluation GREAT guideline development, research priorities, evidence synthesis, applicability of evidence, transfer of knowledge HRP UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction IEC information, education and communication IR implementation research IRP Implementation Research Platform MDG Millennium Development Goal MPH Improving Maternal and Perinatal Health (team) NICHD National Institute of Child Health and Human Development NIH National Institutes of Health NORAD Norwegian Government Agency for Development Cooperation PCC Policy and Coordination Committee PDRH Programme Development in Reproductive Health PFP Promoting Family Planning (team) PMTCT prevention of mother-to-child transmission PUA Preventing Unsafe Abortion (team) RCP Research, Capacity, Policy and Programme Strengthening RCT randomized controlled trial RHR WHO Department of Reproductive Health and Research RHL Reproductive Health Library RP2 Research Project Review Panel SIDA Swedish International Development Cooperation Agency SRH sexual and reproductive health STAG Scientific and Technical Advisory Group TDR Special Programme for Research and Training in Tropical Diseases UN United Nations UNDP United Nations Development Programme UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund WHO World Health Organization HRP External Evaluation 2008−2012 Chapter 6 Page | 187 Abstract Introduction The focus of this case-study is work on implementation research (IR) carried out by the United Nations Development Programme (UNDP)/United Nations Population Fund (UNFPA)/United Nations Children’s Fund (UNICEF)/World Health Organization (WHO)/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP) and WHO Department of Reproductive Health and Research (RHR) between 2008 and 2012, and how this work can be strengthened in the future. IR is research aiming to develop strategies for available or new health interventions, in order to improve access to, and the use of, these interventions by the populations in need. Methods HRP/RHR staff, members of the Scientific and Technical Advisory Group (STAG) and other WHO staff were interviewed. HRP/RHR research studies that have gone through ethical review since 2008 were reviewed, as well as all HRP publications in the same period, pertinent HRP/RHR governance documents (working plans, reports to STAG, annual internal reports) and the strategic plan for 2010–2015. Key findings, comments and conclusions IR has gained attention in recent years, owing to the large underuse of several life-saving interventions in low- and middle-income countries. Since 2009, HRP/RHR has made substantial efforts to raise the priority of IR within the department, and made substantial contributions to include IR as a major topic in funding agencies. Prioritization of implementation research within the department and the World Health Organization HRP/RHR assembled the interteam working group that adopted a common definition of IR, and contributed to the conceptual approach that several WHO departments adopted in 2010. IR was prioritized in the HRP/RHR strategic plan for 2010–2015, including products and activities in progress or to be conducted by each HRP/RHR team. Contribution to implementation research funding HRP/RHR prioritization of IR contributed to the decisions of funding agencies to allocate funds to IR initiatives and specific projects, specifically the Implementation Research Platform (IRP), funded by the Norwegian Government Agency for Development Cooperation, the Department for International Development of the United Kingdom of Great Britain and Northern Ireland, and the Swedish International Development Cooperation Agency; and the IR study of antenatal care in Mozambique, funded by the Flanders International Cooperation Agency. The Implementation Research Platform This initiative has an excellent conceptual approach. It provides funding for IR proposals developed in low- and middle-income countries, with the technical and methodological support of HRP/RHR and the Special Programme for Research and Training in Tropical Diseases (TDR). However, the potential global impact is limited by the relatively low funding and the short time frame, which will only allow small studies at country or subregional level. HRP External Evaluation 2008−2012 Chapter 6 Page | 188 HRP/RHR resources for implementation research HRP/RHR has high-quality resources that should facilitate IR studies: the Reproductive Health Library as a source of effective health interventions and implementation strategies in reproductive health; The WHO guidelines system; The Global Survey; the GREAT Project (Guideline development, Research priorities, Evidence synthesis, Applicability of evidence, Transfer of knowledge). All these resources put HRP/RHR in an advantageous position to design and conduct global IR studies effectively and efficiently. High-quality implementation research studies The eight studies identified are all high-quality IR studies, addressing priority questions. As most of these studies are still ongoing, no impact can be expected at this time. However, the findings of these studies will probably impact public health decision-making in the countries where they are being conducted. Collaboration with other global implementation research studies Providing support to other relevant initiatives, such as the National Institute of Child Health and Human Development’s Global Network Antenatal Corticosteroids Trial, is an efficient way to contribute to answering global IR questions that are being studied by other groups or agencies. Lack of large-scale implementation research studies on priority issues HRP is still not conducting IR studies addressing reproductive priority questions on a large scale. That means designing implementation interventions that may overcome common barriers in several countries; and evaluating them in large-scale studies in several countries. These studies are essential to gain generalizable knowledge in an efficient way and in a relatively short time. Recommendations HRP/RHR should design and conduct IR studies addressing reproductive priority questions on a large scale. Large-scale studies should evaluate strategies for scaling up family planning and improving emergency obstetric health care (i.e. scaling up the use of magnesium sulfate for eclampsia treatment, task-shifting in the provision of health care, integration of onsite contraceptive services with abortion and puerperal care, and scaling up the uptake of emergency contraception). Active participation of HRP/RHR in the IRP should continue. Advocacy to expand future calls for proposals, in order to award larger-scale research projects would be an asset. HRP/RHR should continue the support and participation in other large-scale research studies focusing on low- and middle-income countries, initiated by other agencies or research groups. These collaborative efforts are an efficient use of resources. Setting up a transdisciplinary team of scientists in IR may facilitate HRP/RHR activities in IR. This team should ideally include expertise in the design of implementation strategies, design and conduction of implementation intervention studies, qualitative approaches for assessment of barriers, statistical expertise in design and analysis of IR trials, and behavioural sciences. This team, assembled with either existing or new staff, would work with all thematic teams in a cross-cutting way, providing up-to-date standard methods for implementation science. HRP External Evaluation 2008−2012 Chapter 6 Page | 189 1 Introduction The focus of this case-study is work on implementation research (IR) carried out by the United Nations Development Programme (UNDP)/United Nations Population Fund (UNFPA)/United Nations Children’s Fund (UNICEF)/World Health Organization (WHO)/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP) and WHO Department of Reproductive Health and Research (RHR) between 2008 and 2012, and how this work can be strengthened in the future. 1.1 What is implementation research? In 2010, several WHO departments, agencies and programmes, including HRP/RHR and the Special Programme for Research and Training in Tropical Diseases (TDR), defined IR as research aiming to develop strategies for available or new health interventions, in order to improve access to, and the use of, these interventions by the populations in need (1). This definition agrees conceptually with most of the numerous definitions that exist, including that of the National Institutes of Health (2) and the Canadian Institutes of Health Research. In 2012, this conceptual approach to implementation research was ratified in a commentary published by representatives of the four United Nations (UN) agencies that work on reproductive health, including HRP (3). Both position papers (1, 3) made distinctions between IR, operational research and health systems research. Although the three domains have overlapping areas, the types of questions addressed are different. Operational research aims to develop solutions to current operational problems of specific health programmes, and is very context specific. IR aims to develop strategies for health interventions, in order to improve the use of these interventions by the populations in need. As such, it intends to create generalizable knowledge that can be applied across settings and contexts. Health systems research addresses health system and policy questions that are not disease specific but concern systems problems as a whole (1–3). The three research types require a multidisciplinary approach involving both quantitative and qualitative methods. These distinctions were appropriate and timely, as many definitions for these types of research exist, creating confusion among scientists and decision-makers about the scope, nature, methodologies and issues to be addressed by the research involved (1). We believe that this situation may also be applicable internally to WHO and HRP/RHR more specifically. This chapter will focus on examining what HRP has done since 2008 on IR specifically, and how this can be strengthened in the future. That means reviewing HRP research activities aiming to create generalizable knowledge about strategies to improve the use of effective health interventions by the populations in need. 2 Rationale 2.1 The problem: underuse of life-saving interventions in low- and middle-income countries There is still a large underuse of many well-known life-saving interventions for women and children, particularly in low- and middle-income countries (4). Several of the research studies demonstrating the efficacy of these interventions have been generated by HRP/RHR, or with the collaboration of HRP/RHR. The demand for family planning methods is currently satisfied in only 55% of eligible women; similar rates are shown for an antenatal care (ANC) package in four visits, and skilled birth attendance at birth. It is worth mentioning that these last two are HRP External Evaluation 2008−2012 Chapter 6 Page | 190 actually strategies to provide evidence-based interventions for ANC and childbirth. The rates of use of specific interventions for preventing or treating some emergency obstetric complications are even lower, like the use of magnesium sulfate for preventing or treating eclampsia, or the availability and use of blood products to treat severe obstetric haemorrhage – which, in low-income countries, are only used in tertiary referral hospitals. The use of antenatal steroids in preterm babies, to reduce mortality, shows rates below 10%, and the proportion of children sleeping under insecticide-treated nets to prevent malaria does not exceed 35% (5–8). The body of evidence supporting the use of these interventions has been available for 10–30 years. This gap between knowledge and practice can largely be explained by two factors: weak health systems, with constraints in infrastructure, equipment, supplies and personnel; and lack of effective evidence-based implementation strategies. Regarding the latter, as HRP/RHR and representatives of other UN agencies recently acknowledged, “historically we have placed the overwhelming majority of our scientific emphasis and funding on the ‘what to do,’ – i.e., the development of highly effective interventions for preventing maternal and newborn deaths – and not on the ‘how to do it’” (3). As Richard Grol said 15 years ago, “most approaches to changing health care practices are more often based on beliefs than on scientific evidence. Evidence based practice should be complemented by evidence based implementation” (9). 2.2 Scaling up life-saving interventions There is a wide international agreement on the urgent need for scaling up life-saving interventions as a way to reduce maternal and infant mortality and morbidity. The accomplishment of Millennium Development Goals (MDGs) 4 and 5 largely depends on these efforts, but the concept clearly exceeds the importance of reaching the goals in 2015 and should be sustained in the future as a main objective. Basically (and perhaps a bit schematically), there are two ways of doing this that should be supported in parallel. One is strengthening the health systems; increasing and improving health infrastructure, personnel, supplies and equipment will increase the use of effective interventions and improve the health status of the population. This has been observed in South Asia and some Latin American countries in the last 25 years (10). While there is no doubt that this is a sustainable way, it is worth acknowledging that it is a long-term process. In general, the low-income countries in the most need have the weakest systems; thus, it would take decades to have solid health systems in such countries, while thousands of women and children unnecessarily die during the process. A second way for scaling up is through implementing disease-specific vertical interventions. These interventions can be superimposed on ineffectual health systems, or can bypass them, and achieve improvements in the coverage of beneficial practices and health status in short periods. Examples of such vertical interventions are vitamin A fortification of sugar, polio immunization, child health, conditional cash transfers in Mexico, and the West African Onchocerciasis Control Programme (11, 12). As countries and their health systems improve and develop, these vertical interventions are absorbed and mainstreamed into the regular health system (12). Methodologically, it appears that IR is necessary to develop and evaluate such effective implementation strategies. 2.3 Why has this topic has been included in this evaluation? IR was included in the HRP/RHR agenda only recently, and, consistently with that, it was not previously included in HRP/RHR external evaluations as a separate topic. The 2003–2007 external evaluation included some aspects of implementation research within the chapters HRP External Evaluation 2008−2012 Chapter 6 Page | 191 dedicated to knowledge synthesis and transfer, and maternal and perinatal care. In mid-2008, the Policy and Coordination Committee (PCC) proposed the follow-up action to the 2003–2007 external evaluation of HRP, which included recommendations and proposed action to address the conclusions of the external evaluation. The main overall recommendations related to IR include: • accelerating removal of the remaining barriers for translating evidence from research into practice; • reviewing and strengthening strategies at the interface with the rest of RHR for better knowledge transfer and exchange; • developing strategies to feed the evidence obtained more systematically into the work of cosponsors, donors and partner countries, to maximize efforts to bridge the gap between research and practice; • identifying operational barriers or facilitators to uptake of evidence; • strengthening translational research and capacity building for country partners in this field. IR gained more attention as an independent and growing field in which HRP/RHR showed willingness to invest human and financial resources. As already mentioned, IR was recognized as a component of the strategy to narrow the gap between the MDGs and the slow uptake of clinical research findings into routine care (1, 3). 3 Methods To conduct this external evaluation, the reviewer visited the HRP/RHR offices at WHO Geneva and interviewed staff involved in IR projects. Staff suggestions and shared experiences were incorporated in this document. Experts in IR were also interviewed. The HRP list of projects was revised to identify projects related to IR. Lead officers for each of these projects were contact by e-mail and asked to share relevant information: the final approved protocol, actual state of the project, intermediate and final evaluations, published manuscripts and any additional information that would be useful to get a better understanding of the project. The HRP/RHR web site was used as a source of information to identify activities related to the Implementation Research Platform (IRP). The reviewer also received a list of all HRP/RHR manuscripts published in a peer-reviewed journal from 2008 to 2012. The reviewer examined the list in sequential steps, in order to highlight only those publications that referred to IR. Reviewers were provided with a file containing all HRP/RHR peer-reviewed publications published between 2008 and 2012. The following methodology was applied to obtain only those publications related to IR. First, duplicated titles were deleted. Second, titles related to basic science research, literature reviews, global trends and systematic reviews/meta-analysis were deleted. Third, titles that were clearly not related to IR were deleted. Fourth, abstracts and/or full text from publications potentially related to IR were read. Finally, IR projects and reviews related to this topic were identified. Overall, the main limitations found during the production of this case-study were: • the lack of a clear distinction between HRP IR activities and HRP technical cooperation activities regarding implementation; • difficulty in measuring the outcomes and impact of various projects that are still ongoing; • difficulty in contacting some of the lead investigators by e-mail. HRP External Evaluation 2008−2012 Chapter 6 Page | 192 4 Key findings 4.1 Research governance milestones at HRP related to implementation research As a follow-up to the recommendations of the 2003–2007 HRP/RHR External Evaluation Report (12), and the follow-up actions to these recommendations proposed by PCC in mid-2008, HRP/RHR took several actions in order to prioritize and enable conduction of IR activities. 4.1.1 Interteam working group on implementation research. In 2009, HRP/RHR established the Interteam Working Group on Implementation Research coordinated by Dr Eduardo Bergel (System and Information Service). The working group served as a forum for the development of a collective vision for the department, and facilitation of the mapping and expansion of the work in IR. Although several groups within the department were working on implementation activities, it seems clear that, at that time, there was still no agreement about the scope and focus of implementation, operational, and health systems research. Nevertheless, the working group adopted a definition of implementation, which is one of the most accepted and cited in the literature: Implementation Research is the scientific study of methods to promote the systematic uptake of clinical research findings and other evidence-based practices into routine practice, and hence to improve the quality (effectiveness, reliability, safety, appropriateness, equity, efficiency) of health care. It includes the study of influences on healthcare professional and organizational behaviour (13). The working group also posed several crucial questions to the Scientific and Technical Advisory Group (STAG), in order to get recommendations to guide IR activities: • can STAG identify one or more evidenced-based interventions that should be prioritized for the development and evaluation of implementation/scaling-up strategies? • are there any major issues not addressed by the group? • does STAG agree with the working definition of IR proposed? • which areas of work within IR should be prioritized? The main answers and recommendations from STAG included: • endorsing the working definition of IR, but STAG noted that it ignored other dimensions of health systems (this shows that even inside STAG, the focus of IR was not clear); • recommending raising the profile of IR as a key component for improving reproductive health; • stressing that vertical interventions might improve some outcomes at the expense of other aspects of the health system (again, this showed that a focus on IR was seen as competing with health system research and not complementing it); • merging efforts with other groups like the Knowledge Synthesis and Exchange group. In 2010, following this last recommendation, the Implementation Research and the Knowledge Synthesis and Exchange working groups merged into a single entity. This entity further developed into the GREAT project (see Section 5.2). HRP External Evaluation 2008−2012 Chapter 6 Page | 193 4.2 Implementation research within the HRP strategic plan for 2010–2015 The HRP/RHR strategic plan for 2010–2015 (14) recognized the need for IR to reveal barriers in reproductive health care and its utility to develop appropriate delivery systems for innovative technologies and knowledge. It was proposed to use IR to develop new technologies for the translation of research findings into practice. However, this topic was not considered separately within the cross-cutting areas in the budget description. The Reproductive Health Research budget for 2012–2013 notably highlights several endeavours and projects involving IR. Moreover, in this latest report (14), IR is considered as a cross-cutting area. Having said that, it is difficult to find the exact budget for IR projects, given that several teams within HRP/RHR finance most of them, and the overall IR budget is split into different budget sections. Nevertheless, it is worth mentioning that, according to a graph presented in this report, HRP/RHR will devote approximately U$S 18–19 million and the Programme Development in Reproductive Health (PDRH) will devote approximately U$S 4–5 million in IR projects. In addition, the same report specifically refers to those IR aspects that are planned for implementation in 2012–2013. Those projects clearly involving IR, and their level of priority, are discussed next. Approximately 16 products and activities within the HRP strategic plan for 2010–2015 (14) are related to IR. Among these, seven are considered vital, six essential and two important. The vital products and activities are related to the following areas: • IR research in maternal and perinatal health: − improving community-level maternal and perinatal care; − integration of maternal and newborn health with HIV/malaria and other health system issues in Mozambique; • IR to expand access to medical abortion; • IR to address, in ANC, violence against women; • IR to establish a model and best practice for integrating gender and gender-based violence interventions into national AIDS plans/strategies; • conducting studies of implementation of assessment tools for reproductive morbidity; • IR: support and monitoring of projects funded by the IRP. In addition, to activities categorized as vital, essential activities include: IR to address infertility in health-care settings; IR to increase the use of antenatal corticosteroids for prevention of neonatal mortality; IR on midlevel provision of safe abortion care; and IR to improve reproductive health programmes in different regions across the globe. 5 Special initiatives and projects 5.1 The Implementation Research Platform One of the most innovative initiatives of WHO is the IRP, which is supported by the Government of Norway (Norwegian Government Agency for Development Cooperation, NORAD), the Swedish International Development Cooperation Agency (Sida) and the Department for International Development (DFID) of the United Kingdom of Great Britain and Northern Ireland. The IRP was created to address issues related to health-care delivery and to explore challenges related to the scale-up of effective interventions. Specifically, the IRP supports research that: HRP External Evaluation 2008−2012 Chapter 6 Page | 194 • identifies common implementation problems and their main determinants that hinder effective access to interventions; • develops and tests practical solutions to these problems that are either specific to particular health systems and environments or that address a problem common to several countries in a region; • determines the best way of introducing these practical solutions into the health system and facilitates their full-scale implementation, evaluation and modification as required. In 2010, the IRP launched its first calls for proposals orientated towards the development of partnerships in low- and middle-income countries that are country led and linked to national strategies; involve policy-makers and implementers; build capacity to undertake and lead such research; and establish networks and multicountry collaborations. Seven research projects from Burkina Faso, Guatemala, India, Kenya, the Middle East (Egypt, Lebanon, Occupied Palestinian Territories, Syria), Nepal and Uganda were funded. The addressed topics were: malaria, HIV, emergency obstetrics care, and maternal and child health care. More details can be accessed on the IRP web site (15). Three of the projects are managed in collaboration with HRP/RHR (Guatemala, Middle East, Uganda). They are described in Section 6. A second call for applications was released in July 2012. It is worth mentioning that, although the conceptual and methodological approach is very appropriate, the short time frame and the limited amount of funds (2 funded years; US$ 350,000 per project) do not allow research studies to be designed and conducted across several settings. Moreover, the proposals are generated at country level, which may, in some cases, imply that the problems or solutions addressed have a local or regional nature rather than global implication. 5.2 The GREAT project (guideline development, research priorities, evidence synthesis, applicability of evidence, transfer of knowledge) This new project propose an approach to tackle the knowledge-to-action gap, which includes an integrated process of identification of priority problems, guidelines development and implementation activities. The project is based on the “Knowledge to Action Framework” proposed by Graham in 2006 (16), which is an appropriate platform to consider the intricate process of creation and application of knowledge. The project relies on two solid WHO resources and initiatives: the WHO Reproductive Health Library (RHL), and the WHO guidelines production system. The RHL provides free access to high-quality systematic reviews on sexual and reproductive health (SRH), mainly Cochrane reviews. It is without doubt a reference tool to allow practitioners and policy-makers in low- and middle-income countries to assess the evidence- based interventions that have proven effectiveness to improve reproductive, maternal and perinatal health. In addition, the RHL now also includes systematic reviews of dissemination and implementation strategies, developed by the Cochrane Effective Practice and Organisation of Care Group. The WHO guidelines production system started in 2007 and HRP/RHR is developing evidence- based guidelines in SRH using the GRADE (grades of recommendation, assessment development and evaluation) approach for appraising the quality of evidence and determining the strength of recommendations and following the WHO Guidelines Review Committee standards. Guidelines for the prevention and treatment of postpartum haemorrhage (17) and HRP External Evaluation 2008−2012 Chapter 6 Page | 195 pre-eclampsia/eclampsia were already developed (18). WHO guidance is a unique mandate for setting standards internationally, and low- and middle-income countries very much rely on these standards. Therefore, the GREAT project is a strong resource for providing evidence on which are the best interventions, how priority problems should be managed, and what is known about strategies to effectively implement them in routine practice. However, it is yet not clear how the proposed process will lead to defining global implementation research questions and prioritizing and converting them into research protocols. Neither is it clear how the process will be linked with the funding decisions within HRP/RHR. 5.2 Global survey on maternal and perinatal health This is a research project implemented by WHO periodically in a global network of health facilities. The Global survey 2010 was implemented in 377 facilities from 29 countries (19, 20). This is the largest study ever conducted looking at the prevalence of pregnancy-related complications and severe maternal outcomes using standardized definitions across countries. One of its objectives is to provide information on the rate of use of evidence-based interventions in low- and middle-income countries. One of the current major problems for scaling up effective interventions is that in low-income and some middle-income countries, there are no good information systems that routinely or periodically report the use of key evidence-based interventions using similar definitions. The survey allows researchers to gather data on more than 314 000 pregnant women, 312 000 infants born alive and 4000 stillbirths. The 2010 survey showed that, despite the high coverage of essential interventions, there are variations in terms of health outcomes and indicators of health performance of care. The information coming from the global survey will allow detection global or regionally of underused interventions, and will provide a system to monitor the trends and allow an evaluation of implementation strategies. A series of activities are planned to disseminate the results of the study. Two main manuscripts have been submitted to a peer-reviewed journal and the first 10 secondary analyses are being prepared. 5.3 The WHO Strategic Approach to strengthening sexual and reproductive health policies and programmes This HRP/RHR initiative started more than 15 years ago, to assist countries to undertake a systematic, evidence-based approach to develop, test and scale up a wide range of different complex interventions to improve the reproductive health of communities and individuals (21). The Strategic Approach is a three-stage process. Stage I is a strategic assessment to identify community and programme needs and priorities. Stage II involves investigating, through pilot studies and research, the recommendations for policy change and the community and programmatic interventions to improve access, utilization and quality of care in service delivery. In the third stage of the Strategic Approach, the findings and results from the first two stages are used to scale up interventions for wider impact. The Strategic Approach has today been used by over 35 countries, to address a variety of different reproductive health issues, and in 10 of these the process has been used two or more times to address additional reproductive health issues. Two examples of the Strategic Approach that were carried out in Zambia and Malawi are discussed next. In 2008, a strategic assessment was conducted in Zambia to facilitate the Ministry of Health to build national consensus about strategies and interventions to reduce unwanted pregnancy and unsafe abortion. A qualitative approach was used to interview over 400 persons in urban HRP External Evaluation 2008−2012 Chapter 6 Page | 196 and rural areas. The strategic assessment generated recommendations for improving access and comprehensive care of unintended pregnancies and abortion. Specifically, it is mentioned in the report that the recommendations of the strategic assessment should be implemented on two fronts: addressing the causes of unintended pregnancies; and addressing the barriers to safe abortion. In 2009, a strategic assessment was conducted in Malawi to address the issue of unsafe abortion. This strategic assessment showed that one important barrier in access to abortion services is the restrictive abortion law. The restrictive law does not prevent women from procuring abortions but instead it forces them to seek abortion service clandestinely. While post-abortion care services were available, the assessment showed that the stigmatization of abortion was an important barrier for both the user and the provider. Another finding was that adolescents were particularly compromised with respect to post-abortion care services and other reproductive health services in general. The main barriers and weaknesses identified in the strategic assessment were: inadequate legal framework to address unsafe abortion; weak implementation of recommendations related to unsafe abortion and the prevention of unplanned pregnancies; cultural practices and gender inequities; and unavailability and inaccessibility of adequate family planning commodities and services. In addition, the assessment showed some opportunities, such as: Malawi has demonstrated some level of political will by putting in place several policies and action plans to address maternal mortality rates; generation of tangible evidence on unsafe abortion by the Ministry of Health; and unsafe abortion is easily preventable and would involve relatively inexpensive methods and procedures (complications of unsafe abortions are more expensive to treat). The strategic assessment portrayed a series of recommendations taking into consideration the barriers and weaknesses, as well as the opportunities in Malawi to address the issue of unsafe abortion. It should be mentioned that some activities of the Strategic Approach do not fall under the definition of IR, as they intend to demonstrate a process rather than to answer a research question involving implementation aspects. In addition, a new endeavour was created called ExpandNet. ExpandNet is as a global network of public health professionals and scientists seeking to advance the practice and science of scaling up successful health innovations tested in experimental, pilot and demonstration projects. More details can be accessed on the ExpandNet web site (22). 6 Research projects A total of 160 studies between 2008 and 2012 have been identified that were approved by the Ethics Review Committee (ERC) or Research Project Review Panel (RP2) and thus can be considered research studies. Among them, 25 studies (16%) were identified as having a focus on implementation – either operational or implementation research. The HRP groups involved in the majority of these projects are: Research, Capacity, Policy and Programme Strengthening (RCP; n = 13); Improving Maternal and Perinatal Health (MPH; n = 4); Preventing Unsafe Abortion (PUA; n = 3); and Promoting Family Planning (PFP, n = 1). Thematic areas are distributed in: unsafe abortion, family planning, pre-eclampsia, prenatal care, obstetric emergencies and sexually transmitted infections. More details of these projects are shown in Annex 1. Based on their focus, research questions and methods, it was considered that 8 (5%) studies were actual IR studies. The analysis that follows focuses on these studies. The study areas are maternal and perinatal care and safe abortion. The main aims include increasing skilled birth attendance; improving the quality of maternal, neonatal and emergency obstetric care; improving ANC and antenatal screening of syphilis; and task-shifting for medical abortion. Six are single-country studies conducted in Guatemala, Mongolia, HRP External Evaluation 2008−2012 Chapter 6 Page | 197 Mozambique, Nepal, the United Republic of Tanzania and Uganda and two are multicountry studies conducted in the Middle East region (Egypt, Lebanon, Occupied Palestinian Territories, Syria), and in Latin America (Argentina, Brazil), Africa (Democratic Republic of the Congo, South Africa) and Asia (India, the Philipines, Thailand). Most of the interventions under study are multifaceted, mainly including components orientated toward health-care workers’ capacity building or knowledge transfer, but also involving different models of organization of care and community-level interventions. The design of the interventions has been based on, or refined by, formative research in five of them. All are prospective intervention studies, most of them with experimental designs: cluster randomized controlled trial (RCT; 4); individual randomized trial (1); stepwedge design (1); interrupted time series (1); and a non-randomized intervention study (1). HRP/RHR has different grades of involvement in the studies. There are some projects awarded by the IRP in 2010, in which the principal investigators are from different countries and HRP has provided methodological support and management. In other studies, the original idea was developed at HRP, and the studies were designed in collaboration with country investigators, or vice versa. Two studies are already completed, while six are still ongoing. Each of these eight studies are described next. 6.1 A matched pair cluster-randomized implementation study to measure the effectiveness of an intervention package aiming to decrease perinatal mortality and increase institution-based obstetric care among indigenous populations in Guatemala This is a matched pair cluster randomized trial aiming to evaluate the effectiveness of a package of interventions to increase the prevalence of institution-based care and reduce perinatal mortality in the four districts with the highest maternal mortality ratio in Guatemala. Specifically, the package includes three interventions: (1) to train health-care professionals in emergency obstetric and perinatal care using a high-fidelity, low-tech, in situ, multidisciplinary simulation training curriculum; (2) to design and implement a social marketing strategy that promotes institution-based delivery; and (3) to integrate the role of an obstetric nurse and professional midwife in intervention communities to act as liaisons between traditional birth attendants and public health units. 6.2 A demonstration project for the implementation of the WHO antenatal care model in Mozambique This is a stepwedge design intervention study. The goal of this project is to determine the effect of an intervention designed to increase the use of evidence-based practices included in the ANC (package by midwives and other health professionals) in prenatal clinics in Mozambique. The intervention consists of four different components: (1) a seminar for health- care providers and distribution of printed materials to clinics to increase awareness of the WHO ANC recommendations, the package of interventions included in the Mozambique ANC model and the project objectives; (2) identification of midwives or other health-care professionals interested in participating as project facilitators; (3) a planning workshop on how to implement the ANC package and other training activities; and (4) implementation of the programme at ANC clinics, including the use of reminders for midwives or other health-care providers and patients. HRP External Evaluation 2008−2012 Chapter 6 Page | 198 6.3 Assessing the acceptability, feasibility and effectiveness of a strategy for improving the quality and safety of maternal/neonatal health care in the health system contexts of four Middle East countries This has a quasi-experimental design using a longitudinal prospective interrupted time series (pre–post intervention design with no control). The study has two primary objectives: (1) to test the acceptability, feasibility and effectiveness of a multifaceted quality-improvement strategy combining clinical audit, feedback and engaging opinion leaders with a focus on the management of maternal/neonatal near-misses; and (2) to investigate how differences in the health system context across four Middle Eastern countries identified in public hospitals in Egypt, Lebanon, Occupied Palestinian Territories and Syria, affect the acceptability and feasibility of this quality-improvement strategy. The intervention encompasses different aspects: (1) audit and feedback of maternal/neonatal near-miss cases by collecting information on a prospective basis at equal time intervals; (2) interactive workshops to train health-care providers and hospital administrators on how to conduct criterion-based audit – (a) establishing standards of good practice; (b) measuring current practice; (c) identifying gaps in current practice and feedback concerning these gaps to health professionals; (d) implementing recommendations for change; and (e) re-evaluating practice and making recommendations; (3) training would also standardize the audit and feedback procedures across hospitals; and (4) focus group discussions with junior doctors in each hospital at the conclusion of the intervention. 6.4 Innovations for increasing access to integrated safe delivery, PMTCT and newborn care in rural Uganda This is a community-based, three-arm, non-randomized trial (quasi-experimental study). The primary objectives of the study are: (1) to develop and implement an integrated intervention that includes vouchers for institutional deliveries and care of complications, and home visits by community health-care workers in pregnancy and the postnatal period in four health subdistricts in rural Uganda; (2) to understand the implementation and learning processes used to deal with challenges encountered during the implementation of the intervention; (3) to assess the intervention effects on the proportion of deliveries occurring in health facilities and on the uptake of prevention of mother-to-child transmission (PMTCT) of HIV; (4) to assess the effects of implementation of the intervention on the health system; and (5) to engage stakeholders and disseminate implementation experiences and findings in order to inform policy and scale-up in Uganda. The intervention has three components: (1) use of community health workers (CHWs) for community-based maternal, newborn and PMTCT promotion – (a) CHWs will register women of childbearing age, identify pregnant women and make home visits (before and after delivery to promote proper care for mothers and newborn babies; and (b) CHWs will carry out community mobilization and dialogue to promote community-wide behaviour change; they will also encourage mothers to test for HIV/AIDS during ANC, and for those who disclose their status, they will encourage adherence to the antiretroviral treatment for mothers and babies; (2) provision of vouchers for maternity services and transport to access institutional delivery, and care for maternal postpartum complications and for sick newborn babies; and (3) training, provision of basic drugs and supplies, and integrated support supervision to strengthen health-care facilities. 6.5 A cluster randomized controlled trial to evaluate the effectiveness of the clinically integrated RHL evidence-based medicine course This is a multicentre cluster randomized trial. The main objective is to evaluate whether the RHL-evidence-based medicine (EBM) course is effective in improving knowledge, skills and competencies as compared to passive dissemination of resource materials. The main outcome HRP External Evaluation 2008−2012 Chapter 6 Page | 199 is to assess the gain in EBM knowledge and change in attitudes and skills competence. Participants are training institutions in obstetrics and gynaecology in the participating countries. (1) The RHL-EBM course, which is a clinically integrated e-learning course, will be the experimental intervention; and (2) the second intervention is self-directed learning (passive dissemination of EBM teaching materials. The RHL-EBM course will be compared to passively disseminated EBM resources from the WHO RHL workshop-based course that has the same learning objectives. The study is ongoing in Latin America (Argentina, Brazil), Africa (Democratic Republic of the Congo, South Africa) and Asia (India, Philipines, Thailand) (23). 6.6 Comparison of the safety, efficacy, and feasibility of medical abortion by physicians or non-physicians in Nepal This is a multicentre randomized controlled equivalence trial conducted in five rural districts. It is actually an effectiveness study with a focus on implementation issues (who can provide effective care instead of doctors). The primary objective is to compare the safety, effectiveness and feasibility of medical abortions provided by a physician-led team with those provided by teams of midlevel providers with a referral system. The main outcome was to assess the proportion of complete abortions without manual vacuum aspiration within 30 days of treatment. This project is completed. Findings from this study showed no differences between the proportion of complete abortions assisted by midlevel providers or physician-led teams. One paper has already been published in The Lancet (24). 6.7 The effectiveness of antenatal birth plans in increasing skilled care at delivery and after delivery in rural Tanzania This is a cluster randomized trial conducted in 18 health units in rural areas of the United Republic of Tanzania. The primary objective of the project was to determine the effectiveness of birth plans in increasing skilled care at delivery and after delivery. Intervention units provided ANC with renewed emphasis in birth plans provided by care providers. Control units continued offering ANC as currently provided. Women were interviewed twice, at first encounter and one month after delivery. The main outcome is the proportion of women who sought delivery at the available health units. The project is completed. Two manuscripts have already been published (25, 26) and one is under review. No specific findings were reported to the reviewer prior to the finalization of this case-study. 6.8 Comparison of “one-stop” versus “conventional” service on antenatal syphilis screening in Ulaanbaatar, Mongolia This is a cluster randomized trial at 14 ANC clinics in Mongolia’s capital city. The objectives of the study were: (1) to compare the effectiveness of “one-stop” versus “conventional” service; and (2) to compare the coverage of antenatal syphilis screening between “one-stop” and “conventional” service. In the intervention clinics, the first 2-day workshop was held for ANC providers. The second 2-day workshop was also held for obstetricians and gynaecologists and covered specific knowledge on maternal and congenital syphilis. The intervention clinics were supplied with the necessary materials and supplies. In the control clinics, ANC providers participated in a 2-day training workshop on the project overview, logistics of the project and case-reporting. The obstetricians and gynaecologists received refresher training on the same topics as the intervention group. Treatment of individuals with syphilis and their partners was given free of charge in all facilities. The primary outcomes of the study are (1) utilization of antenatal syphilis screening at the first antenatal visit and at the third trimester of gestation; (2) detected syphilis cases; and (3) the number of congenital syphilis cases. The project was completed but no specific findings were reported to the reviewer prior to the finalization of this case-study (27). HRP External Evaluation 2008−2012 Chapter 6 Page | 200 It is important to note in relation to these studies that, although they are a small proportion of all HRP projects, all are high-quality IR studies. The designs of the interventions have been based or refined by formative research in most studies, and the study designs are rigorous. The focuses of the studies are related to high priority-areas and problems. On the other hand, it is worth mentioning that there are no studies dealing with family planning issues, a high-priority area with unmet needs. Additionally, most of the interventions and studies were designed for single countries. It is likely that the interventions were developed or adapted more for the country context than for a global context. This may prevent easy generalization of the findings, thus limiting the impact of the interventions. 6.9 Collaboration with other agencies or research groups on implementation research initiatives Since 2007 the MPH group of HRP/RHR has supported the Antenatal Corticosteroids Trial designed and conducted by the National Institute of Child Health and Human Development (NICHD) Global Network for Women’s and Children’s Health Research. This study is a cluster randomized trial conducted in 102 health districts in six countries (Argentina, Guatemala, India, Kenya, Pakistan and Zambia), to evaluate whether a complex intervention facilitating the identification of women at high risk for preterm birth and the administration of antenatal corticosteroids increases the use of steroids in preterm babies and reduces neonatal mortality. The support provided by HRP/RHR facilitated the implementation of the trial at country level. 7 HRP peer-reviewed publications 2008–2012 Reviewers were provided with a file containing all HRP peer-reviewed publications published between 2008 and 2012. Over a total of 417 titles, 34 were eligible for full-text review. From these, eight were considered directly related to HRP IR projects (see Annex 2 for more detail). In addition, one more publication was found (26), related to a birth plan IR project in the United Republic of Tanzania, that was not included in the original files. Three papers reported studies started before 2008. Six papers were protocols, preparatory activities or part of the formative research of some of the projects described above. One paper (24), reports the results of the RCT comparing midlevel health-care workers with doctors providing medical abortion services in Nepal. 8 Conclusions IR has gained attention in recent years, due to the large underuse of several life-saving interventions in low- and middle-income countries. Since 2009, HRP/RHR has made substantial efforts to raise the priority of IR within the department, and made substantial contributions to include IR as a major topic in funding agencies. For the future, HRP/RHR has all the capacities to design and conduct large-scale IR projects that will provide answers to scale up priority reproductive interventions at a global level. The main positive findings and some pending issues are presented next. 8.1 Positive findings • Prioritization of IR within the department and at WHO: HRP/RHR assembled the interteam working group that adopted a common definition of IR, and contributed to the conceptual approach that several WHO departments adopted in 2010. IR was prioritized in the HRP/RHR strategic plan for 2010–2015 (14), including products and activities that are ongoing or will be conducted by each HRP/RHR team. HRP External Evaluation 2008−2012 Chapter 6 Page | 201 • Contribution to IR funding: HRP/RHR prioritization of IR contributed to the decisions of funding agencies to allocate funds to IR initiatives and specific projects. The consultation with Norway in 2009 made possible the founding of the IRP, coordinated by the Alliance for Health Policies and Systems Research, TDR and HRP, and funded by NORAD, DFID and Sida. The collaborative work with the Flanders International Cooperation Agency made possible the IR study of ANC in Mozambique. • The IRP: this initiative has an excellent conceptual approach. It provides funding for IR proposals developed in low- and middle-income countries, with the technical and methodological support of HRP/RHR and TDR. The knowledge created through this initiative will be very relevant for the countries or regions included in the funded studies. The potential global impact, however, is limited by the relatively low funding and the short time frame, which will only allow small studies at country or subregional level. • HRP/RHR resources for IR: the department has high-quality resources that should facilitate IR studies. The RHL is a source of effective health interventions and implementation strategies in reproductive health; the WHO guidelines system, which produces high-quality evidence-based guidelines that facilitate scaling up of health interventions in low- and middle-income countries; the Global survey (19), which facilitates a comparable assessment of the use of interventions, and provides a system to monitor trends; and the GREAT project, which provides a conceptual approach to IR challenges. All these resources put HRP/RHR in an advantageous position to design and conduct global IR studies effectively and efficiently. • High-quality IR studies: the eight studies identified in this chapter are all high-quality IR studies, addressing priority questions. As most of these studies are still ongoing, no impact evaluation can be addressed at this time. However, it is expected that the findings from these studies will probably impact public health decision-making in the countries where they are being conducted. One of these studies may have direct implications on regional policies in the Middle East, as it is ongoing in four countries. • Collaboration with other global IR studies: providing support to other relevant initiatives, like the NICHD’s Global Network Antenatal Corticosteroids Trial, is an efficient way to contribute to answering global IR questions that are being studied by other groups or agencies. 8.2 Pending issues • HRP/RHR is not conducting IR studies addressing reproductive priority questions on a large scale yet. Although, some of the HRP/RHR projects took place in different countries of the same region (mainly Asia), HRP/RHR work is primarily orientated towards specific countries’ needs and there is still much work to do regarding interventions that may be useful to overcome common barriers in several countries. Studies that use a global approach are extremely necessary to gain generalizable knowledge in an efficient way and in relatively short periods. HRP/RHR has extensive experience in conducting large-scale studies investigating new interventions in maternal health, family planning, and methods for safe abortion. HRP/RHR have done a good job in creating generalizable knowledge regarding interventions to improve health (such as the AMTSL (Active Management of the Third Stage of Labour) trial (28)), but in the same period they have not done anything comparable regarding IR. Studies of this magnitude and potential impact are much needed and HRP/RHR has the knowledge and resources to lead them. HRP External Evaluation 2008−2012 Chapter 6 Page | 202 HRP/RHR health specialists were asked to list the main interventions that they believe should be scaled up and that should be prioritized in IR studies. The topics considered as priorities to scale up are listed next. 8.2.1 Preventing unsafe abortion • Conducting IR on a large scale to demonstrate the health-systems feasibility of task- shifting in provision of safe abortion care and post-abortion care: HRP/RHR research has shown that nurses, midwives and auxiliary nurses can provide early medical abortion and manual vacuum aspiration as safely and effectively as physicians and that this is acceptable to women. IR can be useful to translate these findings into programmatic action. The role of other cadres of workers like CHWs in playing a supporting role in the provision of safe care also needs research on a larger scale. • IR that looks at ability to simplify protocols for the provision of abortion care (e.g. eliminating the need for routine follow-up care, allowing part of the drugs regimen required for medical abortion (i.e. the misoprostol) to be taken at home after an initial visit to the facility, eliminating routine ultrasound use) are also important in making care more adaptable to women's needs. All of these have been included as recommendations in the new WHO guidance on safe abortion (29). • Larger-scale research to look at programmatic interventions that integrate onsite contraceptive services to abortion and post-abortion care. • IR into financing mechanisms to ensure access for poorer women can also be helpful if overall universal access to safe care is to be achieved. 8.2.2 Family planning • HRP/RHR research has elaborated safe and efficacious regimens for emergency contraception and facilitated more product registration. However, uptake of emergency contraception remains very limited. IR could inform the integration and expanded use of emergency contraception to prevent unplanned pregnancies. • IR for the systematic and universal introduction of contraception during the postpartum or post-abortion period, and under-fives child-care clinics. 8.2.3 Maternal/reproductive care • In addition to the AMTSL trial, the maternal severe morbidity and mortality approach or near-miss that HRP/RHR used in the multicountry survey (19) are amenable to IR. As mentioned by health leaders, some countries are already adopting this strategy (China implemented a sentinel site system based on our form and Peru is also planning a national scale-up). • IR can also be deployed as a follow-up to the systematic reviews on task- sharing/shifting, to help expand family planning coverage and address human resource shortages for other reproductive health-care interventions. 9 Recommendations HRP/RHR should design and conduct IR studies addressing reproductive priority questions on a large scale. Large-scale studies should evaluate strategies for scaling up family planning, safe abortion and improving emergency obstetric health care (i.e. scaling up the use of magnesium sulfate for eclampsia treatment, task-shifting in provision of healthcare, integration of onsite contraceptive services to abortion and puerperal care, and scaling up the uptake of emergency HRP External Evaluation 2008−2012 Chapter 6 Page | 203 contraception). It would be an asset to have at least three ongoing studies in 2015 covering these issues. HRP/RHR should continue to have an active participation in the IRP. Advocacy to expand future calls for proposals in order to award larger-scale research projects would be an asset. HRP/RHR should continue the support and participation in other large-scale research studies focused on low- and middle-income countries, initiated by other agencies or research groups. These collaborative efforts are an efficient use of resources. Setting up a transdisciplinary team of scientists in IR may facilitate HRP/RHR activities in IR. This team should ideally include expertise in the design of implementation strategies, design and conduction of implementation intervention studies, qualitative approaches for assessment of barriers, statistical expertise in design and analysis of IR trials, and behavioural sciences. This team, assembled with either existing or new staff, would work with all thematic teams in a cross-cutting way to provide up-to-date implementation science standard methods. HRP External Evaluation 2008−2012 Chapter 6 Page | 204 References 1. Remme JH et al. Defining research to improve health systems. PLoS Medicine, 2010; 7(11):e1001000. 2. US National Library of Medicines Institutes of Health, Health Services Research Information Central. Dissemination and implementation science (http://www.nlm.nih.gov/hsrinfo/implementation_science.html, accessed 4 January 2013). 3. Peterson HB et al. Preventing maternal and newborn deaths globally: using innovation and science to address challenges in implementing life-saving interventions. Obstetrics and Gynecology, 2012, 120(3):636–642. 4. Hsu J et al. Countdown to 2015: changes in official development assistance to maternal, newborn, and child health in 2009–10, and assessment of progress since 2003. The Lancet, 2012, 380(9848):1157–1168. 5. Black RE, Morris SS, Bryce J. Where and why are 10 million children dying every year? The Lancet, 2003, 361(9376):2226–2234. 6. Bryce J et al. Reducing child mortality: can public health deliver? The Lancet, 2003, 362(9378):159–164. 7. Jones G et al., and the Bellagio Child Survival Study Group. How many child deaths can we prevent this year? The Lancet, 2003, 362:265–271. 8. Darmstadt GL et al. Evidence-based, cost-effective interventions: how many newborn babies can we save? The Lancet, 2005, 365(9463):977–988. 9. Grol R. Personal paper. Beliefs and evidence in changing clinical practice. BMJ, 1997, 315(7105):418–421. 10. Gillespie D et al. Scaling up health technologies. Baltimore, Johns Hopkins Bloomberg School of Public Health, 2007 (http://www.expandnet.net/PDFs/Gillespie%20report- biblio.pdf, accessed 4 January 2013). 11. Madon T et al. Implementation science. Science, 2008, 318:1728–1729 (http://www.asph.org/userfiles/Policy-forum-implementation-science.pdf, accessed 4 January 2013). 12. UNDP/UNFPA/WHO/WORLD BANK Special Programme of Research Development and Research Training in Human Reproduction (HRP). External evaluation 2003–2007. Executive summary. Geneva, World Health Organization, 2008 (http://whqlibdoc.who.int/hq/2008/WHO_RHR_HRP_08.04_eng.pdf, accessed 4 January 2013). 13. Eccles MP et al. An implementation research agenda. Implementation Research, 2009, 4:18 (http://www.implementationscience.com/content/pdf/1748-5908-4-18.pdf, accessed 15 January 2013). 14. WHO sexual and reproductive health medium-term strategic plan 2010–2015 and HRP programme budget 2012–2013. Geneva, World Health Organization, 2011 (WHO/RHR/HRP/11.09) (http://apps.who.int/iris/bitstream/10665/75215/1/WHO_RHR_HRP_11.09_eng.pdf, accessed 15 January 2013). 15. Implementation Research Platform (http://www.who.int/alliance- hpsr/projects/implementationresearch/en/, accessed 15 January 2013). 16. Graham ID et al. Lost in knowledge translation: time for a map? Journal of Continuing Education in Health Professions, 2006, 26:13–24. 17. Mathai M, Gulmezogle AM, Hill S. WHO recommendations for the prevention of postpartum haemorrhage. Geneva, World Health Organization, 2007. HRP External Evaluation 2008−2012 Chapter 6 Page | 205 18. WHO recommendations for prevention and treatment of pre-eclampsia and eclampsia. Geneva, World Health Organization, 2011. 19. WHO multicountry survey of maternal and newborn health 2010–2012. Geneva, World Health Organization. 20. Souza JP et al., for WHOMCS Research Group. The world health organization multicountry study on maternal and newborn health: study protocol. BMC Health Services Research, 2011, 11:286 (http://www.biomedcentral.com/content/pdf/1472- 6963-11-286.pdf, accessed 15 January 2013). 21. World Health Organization. The WHO Strategic Approach to strengthening sexual and reproductive health policies and programmes. Geneva, World Health Organization, 2007 (WHO/RHR/07.7) (http://whqlibdoc.who.int/hq/2007/WHO_RHR_07.7_eng.pdf, accessed 4 January 2013). 22. ExpandNet (http://www.expandnet.net/, accessed 4 January 2013). 23. Kulier R et al. A cluster randomized controlled trial to evaluate the effectiveness of the clinically integrated RHL evidence-based medicine course. Reproductive Health, 2010, 7:8. 24. Warriner IK et al. Can midlevel health-care providers administer early medical abortion as safely and effectively as doctors? A randomised controlled equivalence trial in Nepal. The Lancet, 2011, 377(9772):1155–1161. 25. Magoma M et al. High ANC coverage and low skilled attendance in a rural Tanzanian district: a case for implementing a birth plan intervention. BMC Pregnancy and Childbirth, 2010, 10:13. 26. Magoma M et al. How much time is available for antenatal care consultations? Assessment of the quality of care in rural Tanzania. BMC Pregnancy and Childbirth, 2011, 11:64. 27. Munkhuu B et al. One-stop service for antenatal syphilis screening and prevention of congenital syphilis in Ulaanbaatar, Mongolia: a cluster randomized trial. Sexually Transmitted Diseases, 2009, 36:714–720. 28. Gulmezoglu AM et al. Active management of the third stage of labour without controlled cord traction: a randomized non-inferiority controlled trial. Reproductive Health, 2009, 6:2. 29. World Health Organization, Department of Reproductive Health and Research. Safe abortion: technical and policy guidance for health systems, 2nd ed. Geneva, World Health Organization, 2012. 30. Evaluating the quality of care for severe pregnancy complications. The WHO near-miss approach for maternal health. Geneva, World Health Organization, 2011 (http://whqlibdoc.who.int/publications/2011/9789241502221_eng.pdf, accessed 15 January 2013). 31. Munkhuu B et al. Feasibility of one-stop antenatal syphilis screening in Ulaanbaatar, Mongolia: women and providers’ perspectives. Southeast Asian Journal of Tropical Medicine and Public Health, 2009, 40:861–870. 32. Fiol V et al. Improving care of women at risk of unsafe abortion: implementing a risk- reduction model at the Uruguayan–Brazilian border. International Journal of Gynaecology and Obstetrics, 2012, 118(Suppl. 1):S21–7. 33. World Health Organization, Department of Making Pregnancy Safer. Pregnancy, childbirth, postpartum and newborn care: a guide for essential practice, 2nd ed. Geneva, World Health Organization, 2006. Chapter 6 Page | 206 Annex 1: Implementation research projects undertaken by HRP/RHR between 2008 and 2012 These projects were approved by the Ethics Review Committee (ERC) or the Research Project Review Panel (RP2) and thus can be considered research studies. Project title HRP teama Country Design Primary objective Intervention Main outcome A cluster randomized controlled trial to evaluate the effectiveness of the clinically integrated RHL evidence-based medicine course (23) RCP/MPH Argentina, Brazil, Democratic Republic of the Congo, India, Philippines, South Africa, Thailand Cluster randomized design The main objective is to evaluate whether the Reproductive Health Library evidence- based medicine (RHL-EBM) course is effective in improving knowledge, skills and competencies as compared to passive dissemination of resource materials Participants are training institutions in obstetrics and gynaecology in the participating countries. Intervention 1: RHL-EBM course, which is a clinically integrated e-learning course, will be the experimental intervention. Teaching takes place in the clinical environment and the postgraduate trainee obtains the theoretical knowledge from the interactive e-learning materials, completes assignments and interacts with her/his facilitator throughout the process. There should be one facilitator per group of 5– 10 postgraduate trainees. Intervention 2: self-directed learning (passive dissemination of EBM teaching materials). Gain in EBM knowledge, change in attitudes, skills competence Comparison of the safety, efficacy, and feasibility of medical abortion by physicians or non- physicians in Nepal (25) PUA Nepal Multicentre randomized controlled equivalence trial (five rural districts) To compare the safety, effectiveness and feasibility of medical abortions provided by a physician-led team with those provided by teams of midlevel providers with a referral system. Women were randomly assigned to a doctor or a midlevel provider for oral administration of 200 mg mifepristone followed by 800 μg misoprostol vaginally 2 days later, and followed up 10–14 days later. Complete abortion without manual vacuum aspiration within 30 days of treatment Chapter 6 Page | 207 Project title HRP teama Country Design Primary objective Intervention Main outcome A matched pair cluster- randomized implementation study to measure the effectiveness of an intervention package aiming to decrease perinatal mortality and increase institution-based obstetric care among indigenous populations in Guatemala MPH Guatemala Matched pair cluster randomized trial To evaluate the effectiveness of a package of interventions to increase the prevalence of institution-based care and reduce perinatal mortality in the four districts with the highest maternal mortality ratio in Guatemala The package includes three interventions: (1) to train health-care professionals in emergency obstetric and perinatal care using a high-fidelity, low-tech, in situ, multidisciplinary simulation training curriculum (PRONTO); (2) to design and implement a social marketing strategy that promotes institution-based delivery; and (3) to integrate the role of the obstetric nurse and professional midwife in intervention communities to act as liaisons between traditional birth attendants and public health units. Decrease in the perinatal death rate in intervention versus control clinics A demonstration project for the implementation of the WHO antenatal care model in Mozambique: a cluster randomized controlled trial MPH Mozambique Two-arm, parallel cluster randomized controlled trial To determine the effect of an intervention designed to increase the use of evidence- based practices included in the ANC package by midwives (and other health- care professionals) in prenatal clinics in Mozambique (1) A seminar for health-care providers and distribution of printed materials to clinics to increase awareness of the WHO ANC recommendations, the package of interventions included in the Mozambique ANC model and the project objectives; (2) identification of midwives or other health-care professionals interested in participating as project facilitators; (3) planning workshop on how to implement the ANC package and other training activities; and (4) implementation of the programme at the ANC clinics, including the use of reminders for midwives or other health-care providers and patients. The delivery of selected health-care practices to women attending prenatal care. Selected screening practices: frequency of women receiving screening for: (a) syphilis at least once during pregnancy; (b) HIV at the first ANC visit; (c) HIV at subsequent visits; (d) anaemia; and (e) hypertension. Selected preventive practices: frequency of women receiving: (a) tetanus toxoid; (b) intermittent preventive malaria treatment; (c) iron supplementation; and (d) antiparasitic treatment for anaemia prevention Chapter 6 Page | 208 Project title HRP teama Country Design Primary objective Intervention Main outcome Assessing the acceptability, feasibility and effectiveness of a strategy for improving the quality and safety of maternal/neonatal health care in the health system contexts of four Middle East countries MPH Egypt, Lebanon, Occupied Palestinian Territories and Syria Two-phased study; phase 1: formative research; phase 2: quasi- experimental design using a longitudinal prospective interrupted time-series design (pre– post intervention design with no control) (1) To test the acceptability, feasibility and effectiveness of a multifaceted quality- improvement strategy combining clinical audit, feedback and engaging opinion leaders with a focus on the management of maternal/neonatal near-misses; (2) to investigate how differences in the health- system context across four Middle Eastern countries identified in public hospitals in Egypt, Lebanon, Occupied Palestinian Territories and Syria affect the acceptability and feasibility of this quality- improvement strategy The intervention encompasses different aspects: audit and feedback of maternal/neonatal near-miss cases by collecting information on a prospective basis at equal time intervals; interactive workshops to train health-care providers and hospital administrators on how to conduct criterion-based audit; focus group discussions with junior doctors in each hospital at the conclusion of the intervention. Proportion of cases of maternal and neonatal near-misses that were inappropriately managed using the WHO- developed tool on maternal and neonatal near-miss (29) Chapter 6 Page | 209 Project title HRP teama Country Design Primary objective Intervention Main outcome Innovations for increasing access to integrated safe delivery, PMTCT and newborn care in rural Uganda RCP Uganda Community- based, three- arm non- randomized trial Primary objectives: (1) to develop and implement an integrated intervention that includes vouchers for institutional deliveries and care of complications, and home visits by CHWs in pregnancy and the postnatal period in four health subdistricts in rural Uganda; (2) to understand the implementation and learning processes used to deal with challenges encountered during the implementation of the intervention; (3) to assess the intervention effects on the proportion of deliveries occurring in health-care facilities and on uptake of prevention of mother-to-child transmission (PMTCT) of HIV; (4) to assess the effects of implementation of the intervention on the health system; and (5) to engage stakeholders and disseminate implementation experiences and findings in order to inform policy and scale-up in Uganda The intervention has three components: (1) use of CHWs for community-based maternal, newborn and PMTCT promotion – (a) CHWs will register women of childbearing age, identify pregnant women and make home visits (before and after delivery to promote proper care for mothers and newborn babies; (b) CHWs will carry out community mobilization and dialogue to promote community-wide behaviour change. They will also encourage mothers to test for HIV/AIDS during ANC, and for those who disclose their status, they will encourage adherence to the antiretroviral treatment for mothers and babies; (2) provision of vouchers for maternity services and transport to access institutional delivery and care for maternal postpartum complications and for sick newborn babies; and (3) training, provision of basic drugs and supplies, and integrated support supervision to strengthen the health-care facility. Service delivery: (1) health-care facility utilization for ANC, deliveries and newborn care; (2) quality of ANC and delivery care; (3) rates of testing for HIV in pregnancy; and (4) number of visits by CHWs per mother/baby pair. Health workforce: (1) abstinence rates; (2) attitude; (3) motivation; (4) satisfaction; and (5) performance. Information: (1) completeness of records; (2) use of data for decision-making; (3) workload; and (4) validity of voucher- related records. Medical technology: (1) availability of drugs and supplies and equipment. Financing: (1) effect on allocations to maternity unit from voucher incomes; (2) effect on availability of resources for the unit; (3) incremental cost per supervised delivery; (4) incremental cost effectiveness of the two packages; (5) the proportion of deliveries occurring in health-care facilities; and (6) uptake of HIV testing during ANC The effectiveness of antenatal birth plans in increasing skilled care at delivery and after delivery in rural Tanzania (25, 26)b The United Republic of Tanzania Cluster randomized trial conducted in 18 health- care units in rural settings in the United Republic of Tanzania To determine the effectiveness of birth plans in increasing skilled care at delivery and after delivery Intervention units provided ANC with renewed emphasis in birth plans provided by care providers. Control units continued offering antenatal care as currently provided. Women were interviewed twice, at first encounter and 1 month after delivery. Proportion of women who sought delivery at the available health units Chapter 6 Page | 210 Project title HRP teama Country Design Primary objective Intervention Main outcome Comparison of “one-stop” versus “conventional” service on antenatal syphilis screening in Ulaanbaatar, Mongolia (27, 31) PFP Mongolia Cluster- randomized trial The objectives of the study are: (1) to compare the effectiveness of “one-stop” versus “conventional” service on antenatal syphilis screening in Ulaanbaatar; and (2) to compare the coverage of antenatal syphilis screening between “one-stop” and “conventional” service on antenatal syphilis screening in Ulaanbaatar In the intervention clinics, the first 2-day workshop was held for ANC providers. The second two-day workshop was also held for obstetricians and gynaecologists and covered the specific knowledge on maternal and congenital syphilis. The intervention clinics were supplied with the necessary materials and supplies. ANC providers in the control clinics participated in a 2-day training workshop on the project overview, logistics of the project and case reporting. As with the intervention group, treatment of syphilis cases and their partners was given free of charge. Benzathine benzylpenicillin was supplied for treatment but not facilities for serological tests. The primary outcomes of the study were (1) utilization of antenatal syphilis screening at the first antenatal visit and at the third trimester of gestation; (2) detected syphilis cases; and (3) number of congenital syphilis cases An intervention to improve knowledge on reproductive health and access to services of migrant youth in Ho Chi Minh City, Vietnam RCP Viet Nam Pretest/post- test design To determine if the level of reproductive health knowledge and access to services of migrant youth can be improved through peer education and mobile clinics Peer education carried out as group discussion sessions, each session involving with 10–15 migrant youths. An education cycle consisted of four sessions at weekly intervals, then another cycle would be carried out on other migrant youths throughout the intervention period of 18 months. Mobile clinics from Hung Vuong Hospital made routine visits every 6 weeks to each factory or commune health centre where the study factory was located, when young migrants were informed and encouraged to attend for counselling, examination and treatment. Increase in knowledge on sexual and reproductive health (SRH) of young migrants, and increase in access to reproductive health services. Comparison of pre- versus post-intervention will be made in terms of the percentage of those who have adequate SRH knowledge, and the percentage of those who have good access to reproductive health services Reproductive health promotion for young migrant factory workers in four main districts in Vientiane Capital, Laos RCP The Lao People’s Democratic Republic NA NA NA NA Chapter 6 Page | 211 Project title HRP teama Country Design Primary objective Intervention Main outcome Mejorar la atención a las mujeres en riesgo de aborto inseguro mediante la capacitación y sensibilización de los profesionales y equipos de salud en la frontera Uruguayo-Brasilena (32) PUA Uruguay (1) Before– after design for the hospital that received the intervention; (2) comparison between the hospital that received the intervention and the control hospital (1) To raise awareness of, and to train, health professionals and other support staff about the need to care for women who undergo an abortion; (2) to empower women Two main interventions: (1) to raise the awareness of, and then train, health-care professionals and other support staff about the need to care for women who may want or who have just had an abortion; and (2) to implement, the programme that Iniciativas Sanitarias had successfully deployed in Montevideo and provide an assessment of its impact at that level. Near-misses cases; women admitted to intensive care unit and/or hysterectomies Improving quality of reproductive health services by strengthening linkages with STI/RTI service RCP Viet Nam Pretest/post- test design (1) To improve the capacity of primary health care and reproductive health-care providers in provision of quality reproductive tract/sexually transmitted infection (RTI/STI) services through training and supervision and monitoring; (2) to ensure availability of information, education and communication (IEC) materials and STI supplies; (3) to evaluate the outcomes of programme interventions on improving the quality of reproductive health-care care services Three intervention components: (1) training of health-care staff at the commune and district levels on the provision of quality RTI/STI services; (2) ensuring the availability of supplies (drugs and condoms) and IEC materials at the district and commune level; and (3) strengthening supervision and monitoring the provision of quality care. ( 1) Proportion of health-care providers who are able to provide RTI/STI services; (2) proportion of clients coming for antenatal care (ANC), family planning services or gynaecological examination who receive essential information on STI prevention, risk assessment and required services related to RTI/STI; and (3) client satisfaction with RTI/STI services Chapter 6 Page | 212 Project title HRP teama Country Design Primary objective Intervention Main outcome A quasi-experimental operations research project to provide life- skills-based training and youth- friendly services on SRH among unmarried youthc RCP China Quasi- experimental design To improve the SRH of unmarried migrant youth by providing life-skills-based education and youth-friendly SRH services; to provide policy-makers with a cost-effective, practical and sustainable approach for improving the SRH of unmarried migrant youth, and promoting safe sexual behaviour Unmarried migrant youth in the intervention group will receive life-skills- based training and education together with youth-friendly SRH service (if required) for 6 months. Prior to the intervention to unmarried migrant youth, preparatory work will be conducted: (1) a 1-day advocacy meeting; (2) building up institutional capacity for youth-friendly services; (3) training 12–16 community health workers (CHWs) and family planning service providers; (4) education: preparing training materials; training of trainers on life-skills-based education; and (5) providing youth-friendly services. Regarding service providers: increase of their understanding of providing SRH education and services to unmarried migrant youth; improvement of their working performance. Regarding unmarried youth: percentage of migrant youth using the service provided by the project; increased knowledge about SRH; increased ability to make decisions or protect themselves; increased contraceptive use among sexually active subjects The effectiveness of a patient education programme, a direct referral system – improving contraceptive uptake by patients suffering from medical illnesses PUA, PFP Sri Lanka Two experimental arms in a pretest and post-test quasi- experimental design To introduce a referral system or a patient-education programme at the medical clinic to improve uptake of modern methods of contraception by women suffering from medical illnesses Group 1: referral to the specialist in the family planning clinic of the institution in the form of a referral card. Group 2: displaying of posters highlighting the conditions that require family planning and the importance of it. Such posters will be displayed at the patient waiting areas of the medical clinics. These interventions would be carried out for a 6-month period Uptake of family planning methods by the client Chapter 6 Page | 213 Project title HRP teama Country Design Primary objective Intervention Main outcome Promoting antenatal care services to improve early detection of pre-eclampsia, urban health centres, Mandalay, Myanmar RCP Myanmar A pretest/post- test quasi- experimental design will be applied in this operations research study To determine if training on pre-eclampsia using the updated training modules based on pregnancy, childbirth, postpartum and newborn care would improve the detection and referral of pre-eclampsia by midwives at the urban health-care centres to the tertiary hospital, thereby reducing the complications and consequences of pre-eclampsia The intervention will be provision of training to midwives and lady health visitors using the updated training modules based on the World Health Organization (WHO) guideline, Pregnancy, childbirth, postpartum and newborn care (33) to improve their skills on detection and referral of pre- eclampsia.The training lasted for 2 days. The training activity included lectures, discussions, demonstration and role play. Each training workshop was planned to be conducted for approximately 20 participants. Proportion of pre-eclampsia cases diagnosed by midwives among pregnant women attending ANC; proportion of pre- eclampsia cases among pregnant women attending ANC referred by midwives. It was expected that at the end of the training, the midwives were able to: (1) assess the pregnant women for pre- eclampsia, develop a birth and emergency plan and advise on danger signs; (2) identify risk factors such as first pregnancy, multiple pregnancy; mention common maternal complications and serious fetal outcomes of pre-eclampsia; when to refer a case of pre-eclampsia; (3) measure blood pressure accurately; and (4) check urine protein accurately Promotion of reproductive health of adolescent migrants in Mandalay City RCP Myanmar Pretest/post- test design without a control population (1) To assess the baseline level of knowledge, perception and practices about reproductive health in adolescent and youth migrant factory workers; (2) to provide reproductive health information and referral information for reproductive health problems, to adolescent and youth migrant factory workers; (3) To reassess the level of knowledge, perception and practices about reproductive health in adolescent and youth migrant factory workers after 12 months of selected interventions; (4) to identify the factors that facilitate and the factors that hinder the success of the intervention; and (5) to contribute to the evidence base related to the intervention model for promotion of reproductive health among adolescent and youth migrants Twenty youths from Red Cross Society will be given a 10-day training on adolescent reproductive health education and counselling and 10 general practitioners (GPs) from the project township will be given a 7 -day training on management of common reproductive health problems and adolescent reproductive health counselling. Trained youth health communicators from the Red Cross Society will give three rounds of small-group heath communication sessions about reproductive health to youth aged 15 and 24 years, in each of 11 participating factories in 18 months. Participating GPs will provide diagnosis, treatment, counselling and referral for adolescent reproductive health problems of youth migrant factory workers at their clinics. The significance of change in knowledge, perception and practice and access to reproductive health information and services from the baseline to the end-line of the intervention Chapter 6 Page | 214 Project title HRP teama Country Design Primary objective Intervention Main outcome Expanding access to sexual and reproductive health information and services for factory migrant youth in Bangkok, Thailandd RCP Thailand Pretest/post- test design To test a model of migrant youth peer educators in communicating and providing SRH information, skills, and counselling services in the community. To provide knowledge and access to SRH information and services to migrant youth Seven-day training workshop for peer educators that comprised both theoretical and practical sessions, covering knowledge on reproductive health physiology, pregnancy, abortion, contraception, STIs including HIV/AIDS, life-skill development, and communication skills. Training planned to be conducted every 6 months for the trained peer educators and new recruited educators involved in the project. Altogether, there will be three training workshops held during the 18 months of intervention. A 5- day training workshop for health staff including topics on adolescent reproductive health. Outcomes: (1) increased knowledge on SRH of migrant youth; (2) increased skills in negotiating desired sexual outcomes; (3) increased access to reproductive health services; and (4) improvement of reproductive health behaviour Pharmacy provision of misoprostol for post-abortion care and other reproductive services PUA Kenya Controlled randomized trial (1) To determine whether training of pharmacists and pharmacy workers leads to improved information provision and/or referral of misoprostol users; (2) to determine whether training of pharmacists and pharmacy workers leads to improved information provision and referral for post- abortion care services; (3) to determine whether linking pharmacists and pharmacy workers and reproductive- health-care providers at the community level leads to an increase in referrals for treatment of incomplete abortion and other reproductive health services; and (4) to utilize the intervention findings to mobilize the Ministry of Health, Pharmacy Board and other partners to develop a policy on the role of pharmacies in the provision of information on misprostol for post-abortion care and reproductive health services Pharmacists and pharmacy workers in the intervention group will participate in an 8- hour training (including the legality of post-abortion care and abortion in Kenya and information on misoprostol for post- abortion care, particularly the correct usage of misoprostol and contraceptive service provision, information, and referral); receive IEC materials and follow- up supportive supervision visits. Pharmacists and pharmacy workers in the control group will not receive this training. Trained simulated clients will be used to complement data collected by pharmacists and pharmacy workers post- intervention. All participants will be fully trained after the data-collection phase. The training of the control group will, however, be advised by the success of the study – if the intervention is effective, then the investigators will provide training to the control group. Increase information/education on post- abortion care misoprostol; increased referrals for post-abortion care; and increased post-abortion contraceptive use Chapter 6 Page | 215 Project title HRP teama Country Design Primary objective Intervention Main outcome Improving early medical abortion service delivery: is a take-home checklist and pregnancy test a safe and acceptable alternative to a routine follow-up visit? PUA Ghana and Zambia Non-inferiority active- controlled randomized clinical trial Overall objective: to obtain evidence on the safety, feasibility and acceptability of a simplified medical abortion protocol consisting of a pregnancy test and checklist for self-referral for follow-up care. Primary objective: to compare the safety of a pregnancy test and checklist for self-referral follow-up care in place of a routine follow-up visit for all women The intervention consists of replacing the follow-up visit at 10–14 days with a follow- up system consisting of a checklist of questions and a pregnancy test to assess the outcome of the abortion and to screen for ongoing pregnancies. Primary end-point: safety. Women will be classified as: complete abortion, ongoing pregnancy, continuing excessive bleeding or other problem (incomplete abortion) Increasing usage and correct application of the partograph in three maternity hospitals in Kabul, Afghanistan RCP Afghanistan Pretest/post- test design (1) To assess the current situation regarding the percentage prevalence and correct usage of the partograph; (2) to determine whether an intervention package addressing the gaps and limitations found in the baseline study will result in increased and correct application of the partograph; (3) to assess the skills of staff Completed partograph training for all staff of all three sites. Regular external supervision and monitoring from the research department The partograph was attached in the patient's record in 87% of cases, but only 20% were used correctly; 24% of the staff expressed the opinion that although the partograph is a good tool, it is impossible to use because they do not have time to fill it in. Doctors stated that most decisions are made based on clinical findings and examination and the midwives said that they are made in consultation with the physician, based on clinical findings, rarely based on the partograph. Most mentioned there is a rush of patients, limited number of staff, insufficient time and staff not trained in the use of the partograph. The following recommendations were made: training, supervision and monitoring of midwives and junior doctors; maternity hospitals should have three shifts with an equal number of staffs in each shift; logistical problems identified should be addressed; patronage and advocacy at the policy-makers’ level; develop good record- keeping practice, possibly computerized NA, no information available. a WHO teams: MPH, Maternal and Perinatal Health; PFP, Promoting Family Planning; PUA, Preventing Unsafe Abortion; RCP, Research, Capacity, Policy and Programme Strengthening. b Magoma M et al. The effectiveness of birth plans in increasing use of skilled care at delivery and postnatal care in rural Tanzania. Journal of Tropical Medicine and International Health, (submitted June 2012 – under revision). Chapter 6 Page | 216 c The project “A quasi-experimental operations research study to provide life-skills based training and youth-friendly service on SRH among unmarried migrant youth in Shanghai” is the same project as “A quasi-experimental operations research project to provide life-skills based training and youth-friendly services on SRH among unmarried youth”. d The project “Expanding access to sexual and reproductive health information and services for factory migrant youth in the Greater Mekong subregion” is the same project as “Expanding access to sexual and reproductive health information and services for factory migrant youth in Bangkok, Thailand”. Other notes No information was sent to reviewers regarding the project “Evaluation of the impact of continuous education programme for health providers and for the community, on quality of care and utilization of reproductive health services in Paraguay”. The project “Training midwives in Kyrgyzstan to provide safe abortion care with mifepristone and misoprostol” is being revised for possible resubmission to the Research Project Review Panel (RP2). Is not yet even approved by RP2 and hence reviewers did not have access to the protocol. The project “Challenges and opportunities for integration of sexual and reproductive health and HIV/AIDS services: operations and implementation research on the cooperation of programmes and services in Peru” was approved by RP2 but there is not enough funding to implement it yet. HRP External Evaluation 2008−2012 (DRAFT) Chapter 6 Page | 217 Annex 2: Publications in peer-reviewed journals related to implementation research projects This table was created using a file containing all HRP/RHR peer-reviewed publications published between 2008 and 2012 that was provided to reviewers. Procedure Number of publications Total number of publication in the files sent to reviewers 417 Number of publication after deleting duplicates 388 Number of publication after deleting titles related to basic science research, literature reviews, global trends and systematic reviews/meta-analysis 145 Number of publications after deleting titles clearly not related to IR 34 IR projects 8 Publications in peer-reviewed journals relating directly to HRP implementation research projects • Costa ML, Cecatti JG, Milanez HM et al. Audit and feedback: effects on professional obstetrical practice and healthcare outcomes in a university hospital. Acta Obstetricia et Gynecologica Scandinavica, 2009, 88:793–800. • Huntington D et al. Impact of a service provider incentive payment scheme on quality of reproductive and child-health services in Egypt. Journal of Health Population and Nutrition, 2010, 28:273–280. • Kulier R et al. A cluster randomized controlled trial to evaluate the effectiveness of the clinically integrated RHL evidence -based medicine course. Reproductive Health, 2010, 7:8. • Magoma M et al. High ANC coverage and low skilled attendance in a rural Tanzanian district: a case for implementing a birth plan intervention. BMC Pregnancy and Childbirth, 2010, 10:13. • Munkhuu B et al. Feasibility of one-stop antenatal syphilis screening in Ulaanbaatar, Mongolia: women and providers’ perspectives. Southeast Asian Journal of Tropical Medicine and Public Health, 2009, 40:861–870. • Munkhuu B et al. One-stop service for antenatal syphilis screening and prevention of congenital syphilis in Ulaanbaatar, Mongolia: a cluster randomized trial. Sexually Transmitted Diseases, 2009, 36:714–720. • Tu XW et al. Long-term effects of a community-based program on contraceptive use among sexually active unmarried youth in Shanghai, China. Journal of Adolescent Health, 2008, 42:249–258. • Warriner IK et al. Can midlevel health-care providers administer early medical abortion as safely and effectively as doctors? A randomised controlled equivalence trial in Nepal. The Lancet, 2011, 377:1155–1161. • Magoma M et al. How much time is available for antenatal care consultations? Assessment of the quality of care in rural Tanzania. BMC Pregnancy and Childbirth, 2011, 11:64.* * This publication was sent separately from the list of publications but was included as it is related to a project included in this case-study. HRP External Evaluation 2008−2012 (DRAFT) Chapter 6 Page | 218 Page | 219 Chapter 7 The status of, and opportunities for strengthening, engagement with the private sector and civil society International Development Team Pricewaterhouse Coopers SA, Geneva, Switzerland Page | 220 This chapter was written by PricewaterhouseCoopers SA (PwC) only for information purposes of HRP/WHO and does not constitute part of any engagement for a tax and/or legal consultancy service. It is not designed to provide tax or legal advice. This document does not necessarily deal with every important topic or cover every aspect of the topic with which it deals. Facts, considerations and findings in this document are based on limited information received from HRP/WHO, which is available in the public domain or is known to PwC other than by reason of the execution of an engagement. PwC did not independently verify any such information. This document is not intended to be relied on by the addressee or any third party. Therefore, this document shall have no binding effect and PwC accepts no liability of any kind for any consequences that might be suffered by the addressee or any third party acting in reliance on this document. Page | 221 Contents Abstract . 223 Introduction and scope of work . 223 Key recommendations . 223 Regular monitoring of engagement with the private sector and civil society . 224 Structures for engaging with the private sector and civil society . 224 The way forward . 224 1 Background . 226 2 Scope of work . 226 3 Approach and methodology . 227 3.1 Approach . 227 3.2 Methodology. 227 4 HRP’s governance structure . 227 5 HRP’s key drivers for engaging with the private sector and civil society . 229 6 HRP’s existing collaborations with the private sector and civil society . 230 6.1 Identification of new products or technologies . 233 6.2 Dissemination of information, tools and policies . 233 6.3 Management of research projects . 233 6.4 Fundraising through commercial activities. 233 7 Findings from analysis of existing collaborations on the basis of the key drivers identified . 234 7.1 Collaborations focused on the identification of new products or technologies . 234 7.2 Collaborations aiming to disseminate information, tools and policies . 235 7.3 Collaborations based on the management of research projects . 235 7.4 Collaborations aiming to carry out fundraising through commercial activities . 236 8 Conclusions and recommendations . 237 8.1 Engagement with the private sector and civil society. 237 8.2 Regular monitoring of engagement with the private sector and civil society . 238 8.3 Structures for engaging with the private sector and civil society . 239 9 The way forward . 240 9.1 Engagement with the private sector and civil society. 240 9.2 Regular monitoring of engagement with the private sector and civil society . 240 9.3 Structures for engaging with the private sector and civil society . 241 References . 242 Appendix 1: PwC approach and methodology . 243 Page | 222 Abbreviations HRP UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction PCC Policy and Coordination Committee NGO nongovernmental organization PEEC PCC External Evaluation Committee PwC PricewaterhouseCoopers SA STAG Scientific and Technical Advisory Group UN United Nations UNDP United Nations Development Fund UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund WHO World Health Organization HRP External Evaluation 2008−2012 Chapter 7 Page | 223 Abstract Introduction and scope of work This report presents the findings, conclusions and recommendations that result from the case- study carried out by PricewaterhouseCoopers SA (PwC), within the framework of the United Nations Development Fund (UNDP)/United Nations Population Fund (UNFPA)/ United Nations Children’s Fund (UNICEF)/World Health Organization (WHO)/World Bank) Special Programme of Research, Development and Research Training in Human Reproduction (HRP) external evaluation 2008–2012. The study focuses on “the status of, and opportunities for strengthening, engagement with the private sector and civil society”. PwC agreed with HRP to undertake the following tasks: • to analyse the HRP Programme and its structure from a governance perspective and within the broader framework of WHO; • to map existing relationships between HRP, the private sector and organizations representing civil society; • to assess the role of key stakeholders (private sector/civil society) and their involvement in HRP’s activities; • to assess existing relationships with both the private sector and civil society, within the framework of WHO, in order to identify legal limitations and room for opportunities; • to identify best practices and opportunities to strengthen engagement with the private sector and civil society. The work was undertaken in October, November and December 2012. Key recommendations • Engagement with the private sector and civil society • To strengthen engagement with the private sector and civil society, HRP needs to develop a clear vision, accompanied by a detailed strategic plan to realize it. This vision and strategy should stipulate HRP’s specific goals for engagement with the private sector and civil society. • HRP can consider the possibility of agreeing on a set of criteria for selection of the type of collaborations that would best serve its needs, and in order to assess new opportunities. A more structured process would increase transparency and accountability. • In order to become a key platform for sharing best practices, evidence and cost-effective interventions, HRP should strive to develop and strengthen a sense of community among all stakeholders dedicated to research on human reproduction. Accordingly, HRP should consider the possibility of arranging a global annual meeting, also inviting interested parties from the private sector and civil society. • To better meet the needs of the private sector and civil society, and in order to promote and ensure easy and wide access to best practices, HRP should consider further developing the existing web site to make it a live web platform (e-discussion and social media). • HRP should appoint an individual to act as a focal point, who would work closely with the Programme director, in order to facilitate and further leverage relations with civil society organizations, particularly with the international nongovernmental organizations HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 224 with which the Programme collaborates on a regular basis and at different levels. Moreover, HRP should consider the possibility of adopting framework agreements with these organizations. Regular monitoring of engagement with the private sector and civil society • HRP should consider the possibility of adopting a performance framework when engaging with the private sector and civil society (objectives, indicators and targets). • HRP should start immediately gathering all financial and nonfinancial information regarding the collaborations with the private sector and civil society. Structures for engaging with the private sector and civil society • Alternative options regarding the best structure to achieve HRP’s strategic goals related to the private sector and civil society may be explored through a feasibility study. • For research projects, HRP should continue directly to engage with the private sector and civil society through its current mandate and executing agency (WHO). • When products or technologies resulting from the research projects are proven to be effective for use in the field, and can be commercialized by a private sector collaborating party, HRP should make sure that royalties are properly negotiated and that funds received are properly monitored and reinvested. • If the collaborating party is a civil society organization entering into commercial agreements with the private sector for the production and commercialization of a new product or technology, HRP needs to ensure that the collaborative agreement (i) provides for public sector pricing to ensure availability of new products or technology in low- and middle-income countries, and (ii) ensures a fair compensation is received, based on the investment made for development of the new product or technology. The way forward The recommendations made regarding the next steps are presented under the headings that follow. Engagement with the private sector and civil society • HRP should develop a concept paper for discussion, outlining the vision for strengthening the engagement with the private sector and civil society and describing related strategic goals. • Following consultation with the Policy and Coordination Committee, HRP should consider the possibility of organizing a workshop to further develop, discuss and validate its vision for engaging with the private sector and civil society, and the related strategic goals. • HRP’s management should further discuss the possibility of convening an annual global meeting on reproductive health, in order to define goals and objectives. • HRP’s management should appoint an individual to act as a focal point and work closely with the director in order to coordinate the above and facilitate relations with all stakeholders, including the private sector and civil society. • HRP should start discussing with the WHO legal office the possibility of adopting framework agreements with those organizations from civil society that collaborate with HRP on a regular basis, and with different HRP teams. HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 225 Regular monitoring of engagement with the private sector and civil society • HRP should work on the development of a monitoring and evaluation framework. • In parallel to the above, HRP should create a database, in order to gather all the financial and nonfinancial information regarding the collaborations with the private sector and civil society. Structures for engaging with the private sector and civil society • HRP should carry out a study to identify the optimal structure to launch new innovative mechanisms and initiatives that aim to raise funds through commercial activities. HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 226 1 Background The Programme of Research, Development and Research Training in Human Reproduction (HRP) was established by the World health Organization (WHO) in 1972, and became a WHO/United Nations Development Fund (UNDP)/United Nations Population Fund (UNFPA)/World Bank cosponsored Special Programme in 1988. The Programme is a global programme of international technical cooperation to promote, coordinate, support, conduct and evaluate research in human reproduction, with particular focus on the needs of low- and middle-income countries. WHO is the executing agency. Over the years, the work of HRP has evolved in order to best respond to its mandate and adapt to evolving needs, global context and commitments. In order to ensure its effectiveness and efficiency in carrying out its mandate, HRP has been subject to periodic independent external evaluations, commissioned by the Programme’s Policy and Coordination Committee (PCC). These evaluations are carried out in order to ensure the efficiency and accountability of the Programme, as well as to advocate for the Programme’s work and respond to the specific requirements of its donors and cosponsors. Within this context, PricewaterhouseCoopers SA (PwC) has been requested to take part in the HRP external evaluation 2008–2012. This evaluation aims to provide information on: (i) the relevance and fulfilment of HRP’s objectives; (ii) its efficiency and effectiveness; (iii) its comparative advantage; and (iv) the impact and sustainability of its work. In doing so, it will provide information that is credible and will enable HRP to continue the incorporation of lessons learnt into the decision-making process of both the Programme and its governing bodies. The HRP external evaluation 2008–2012 consists of an introductory, overview chapter followed by six chapters that each provide an in-depth study of a series of selected topics. Each study has been carried out by an independent consultant reporting to a subcommittee of PCC, entitled the External Evaluation Committee (PEEC). Within the framework of the evaluation, PwC has been requested to perform an in-depth study on “The status of, and opportunities for strengthening, engagement with the private sector and civil society”. The aim of this specific study is to review the collaboration between HRP, the private sector and civil society to date, and to present the lessons learnt, conclusions and recommendations that can be used as a basis to strengthen HRP’s engagement with these sectors. This report first delineates the scope of work and describes the approach and methodology used, and subsequently delves into the HRP’s governance structure and a review of the collaborations to date. The main findings, and the conclusions and recommendations these lead to, are presented in the remainder of the report. 2 Scope of work To carry out this in-depth study, the following tasks were agreed with HRP, to be included in the terms of reference, as follows: • internal mapping: analysis, from the perspective of collaborating with the private sector and civil society, of the HRP Programme and its structure from a governance perspective and within the broader framework of WHO; • external mapping: analysis of existing relationships between HRP, the private sector and organizations representing civil society; HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 227 • analysis of key stakeholders: assessment of the role of key stakeholders (private sector/civil society) and their involvement in HRP’s activities; • WHO context analysis: assessment of the existing relationships with both the private sector, and civil society within the framework of WHO, in order to identify legal limitations and room for opportunities; • final assessment: identification of best practices and room for opportunities to strengthen engagement with the private sector and civil society; • final report: “The status of, and opportunities for strengthening, engagement with the private sector and civil society” (the study). The terms of reference were discussed further during the “kick-off” phase, with key HRP representatives, and it was agreed to focus the study on collaborations aimed at establishing bilateral relationships with HRP to contribute, directly or indirectly, to the implementation of its core activities. In the context of this study, the term “private sector” refers to private enterprises that pursue a commercial and financial interest and that will contribute to HRP’s efforts in the field of reproductive health research. By civil society, the report refers to not-for-profit nongovernmental organizations (NGOs), which in this case have a particular interest in entering into a collaborative initiative with HRP on the basis of a particular research project. 3 Approach and methodology 3.1 Approach The team adopted the following three-phase approach to carry out the study: • phase 1: kick off – agree on the terms of reference and scope of work; • phase 2: perform – data gathering; • phase 3: deliver – data analysis and production of the report. 3.2 Methodology The approach used in the study was to: • enhance understanding of HRP from a governance perspective; • identify key HRP drivers for engaging in collaborations with the private sector and civil society; • review the existing collaborations between HRP and the private sector and civil society; • analyse existing collaborations on the basis of the key drivers identified; • draw conclusions and propose recommendations for the way forward. For a more detailed description of the study approach and methodology, please refer to Appendix 1, “PwC approach and methodology”. 4 HRP’s governance structure In order to enhance understanding of HRP’s governance structure and its current legal status, a review was conducted of the existing HRP–WHO Memorandum of Understanding (HRP- WHO) (1), which establishes the Programme within WHO. The Programme is composed of two governing bodies and one group, namely PCC, the standing committee and the Scientific and Technical Advisory Group (STAG). HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 228 The governing body of the Programme is PCC, which has the following key functions: • to review and decide upon the planning and execution of the Programme; • to review and approve the plan of action and budget for the coming financial period; • to review proposals and approve arrangements for the financing of the Programme; • to review proposed longer-term plans of action and their financial implications; • to review the annual financial statements submitted by the executing agency, and the audit report submitted by the external auditor of the executing agency; • to review periodic reports that will evaluate the progress of the Programme towards the achievement of its objectives; • to review and endorse the selection of members of STAG by the executing agency; • to consider such other matters relating to the Programme as may be referred to it by any cooperating party. PCC, through its membership, represents the following HRP’s stakeholders: • the largest financial contributors; • countries elected by the WHO regional committees; • other interested cooperating parties (two members elected by PCC and others approved as observers by the executing agency); • permanent members (cosponsors and the International Planned Parenthood Federation). The standing committee represents the cosponsors and has the following key functions: • to review plans of action and budget for the coming period, prepared by the executing agency (WHO) and reviewed by STAG, for presentation to PCC (at its annual session); • to make proposals to PCC for the financing of the Programme for the coming period; • to review the reallocation of resources during a financial period; • to provide recommendations to PCC on particular aspects of the Programme, upon request. STAG represents the broad range of biomedical and other disciplines required for the Programme’s activities. Its key functions consist of: • the review, from a scientific perspective, and the content, scope and dimensions of the Programme, including the research areas covered and the approaches to be used; • recommendations on the priorities within the Programme; • continuous and independent evaluation of the scientific and technical aspects of all activities of the Programme to PCC and the standing committee; • review of the plans of action and the budget for financial periods within the scientific and technical component of the Programme. The executing agency (WHO) appoints the director of the Programme and its staff. The director is responsible for the overall scientific and technical development and operation of the Programme, including its plans of action and budget. As WHO is the executing agency of HRP, WHO rules and procedures are applied for any collaboration to be established by the Programme with both the private sector and civil society. HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 229 5 HRP’s key drivers for engaging with the private sector and civil society Key drivers relate to both HRP’s main incentives and its requirements for engaging in collaborations with the private sector and civil society. These were extracted from core HRP and WHO documents, including HRP–WHO’s Memorandum of Understanding (1998, revised in 2012) (1), HRP programme budget for 2012–2013 (including medium-term strategic plan for 2012–2015) (2); the United Nations (UN) Secretary-General’s Global strategy for women’s and children’s health (2010) (3); WHO’s Guidelines on working with the private sector to achieve health outcomes (4); and the WHO policy framework for engaging and working with the commercial private sector (5). The key drivers presented next were set out. • Alignment with HRP's key goal of “promoting, coordinating, supporting, conducting and evaluating research in human reproduction, with particular reference to the needs of developing countries” (1, 2), by: - promoting and supporting research aimed at finding and developing safe and effective methods of fertility regulation and identifying and eliminating obstacles to such research and development; - identifying and evaluating health and safety issues associated with fertility regulation technology, analysing the behavioural and social determinants of fertility regulation, and testing cost-effective interventions to develop improved approaches to fertility regulation within the context of reproductive health services; - strengthening the training and research capabilities in low- and middle-income countries in the field of human reproduction; - establishing a basis for collaboration with other programmes engaged in research and development in human reproduction, which include identification of priorities across the field and coordination of activities in the light of such priorities. • Contribution towards the Secretary-General’s call for action to “generate and synthesize research-derived evidence, and provide a platform for sharing best practices, evidence on cost-effective interventions and research findings” (3): - HRP is the only programme within the UN system addressing research and sexual reproductive health; the Secretary-General is relying on HRP to make a primary contribution. • Contribution to WHO call for “National research capacity [to be] strengthened as necessary, and new evidence, products technologies, interventions and delivery approaches of global and/or national relevance [to be] available to improve maternal, newborn, child and adolescent health” (6). • Promotion of the wide availability of new health-related products to the public at large (WHO legal office interview 21 November 2012). • Promotion of availability of products to the public sector of low- and middle-income countries in sufficient quantities to meet the demand, and at an affordable, or at least preferential price (WHO legal office interview 21 November 2012). • Negotiation, collection and reinvestment of royalties when new products or technologies can generate revenues from their commercialization (WHO legal office interview 21 November 2012). • “Compliance with the WHO principles, policies and guidelines on working with the private sector to achieve health outcomes” (4, 5), as follows: HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 230 - the main objective of the interaction with the private sector should be to further WHO’s mission and policies; - WHO's reputation and values must be ensured; - scientific validity must not be compromised; - real and perceived conflict of interest, either for the staff or for the work of the organization, needs to be disclosed; - relationships with commercial enterprises whose activities are incompatible with WHO’s work, such as the tobacco and arms industries, need to be avoided; - the interaction must contribute to improving public health. Public health gains should be commensurate with the time and expense involved in establishing and maintaining the relationship; - clearly written letters or agreements indicating the contribution (financial and otherwise) that each of the parties brings to the relationship should be the basis for the establishment of the relationship; - commercial enterprises working with WHO must conform to WHO public health policies in the areas of food safety, chemical safety, ethical promotion of medical drug products, tobacco control, and others; - evaluation criteria similar to those already in use by a range of public agencies in assessing potential partnership with commercial enterprises, including the public image, and financial stability and integrity of the company, should be applied. It is noted that the proposed drivers represent a set of principles that can be further discussed and defined by HRP on the basis of additional internal consultations. 6 HRP’s existing collaborations with the private sector and civil society Existing collaborations1 and supporting documents were analysed to identify patterns and trends, while also seeking to make a distinction between collaborations with the private sector and civil society. This exercise enabled the documents to be compared and contrasted, in order to identify differences and similarities. Please refer to Table 1, “Analysis of existing collaborations on the basis of the key drivers identified”. On the basis of the analysis, the existing collaborations were regrouped into four main categories that share common key goals, themes or activities. In addition, the existing collaborations were further examined to see the extent to which they were primarily relying on the private sector or civil society. 1 For confidentiality reasons, the names of the other parties that collaborate or collaborated with HRP are not disclosed. Chapter 7 Page | 231 Table 1 Analysis of existing collaborations on the basis of the key drivers identified Key drivers Collaborations 1. Alignment with HRP’s key goal of “promoting, coordinating, supporting, conducting and evaluating research in human reproduction, with particular reference to the needs of developing countries” (1, 2) 2. Contribution towards the Secretary-General's call for action to “generate and synthesize research- derived evidence, and provide a platform for sharing best practices, evidence on cost- effective and safe interventions and research findings (3) 3. Contribution to WHO call for “National research capacity [to be] strengthened as necessary, and new evidence, products technologies, interventions and delivery approaches of global and/or national relevance [to be] available to improve maternal, newborn, child and adolescent health (6) 4. Promotion of the wide availability of new health- related products to the public at large 5. Promotion of availability of products to the public sector of low- and middle- income countries in sufficient quantities to meet the demand, and at an affordable, or at least preferential price 6. Negotiation, collection and reinvestment of royalties when new products or technologies can generate revenues from their commercialization 7. “Compliance with the WHO principles, policies and guidelines on working with the private sector to achieve health outcomes” (4, 5) 1. Identification of new products or technologies • The main aim of this kind of collaboration is to carry out or evaluate research • Generate and synthesize research- derived evidence • Do not provide a platform for sharing best practices • Do not contribute to national research- capacity strengthening • New evidence, products, technologies, interventions and delivery approaches of global or national relevance are often the result of the collaboration • New products and technologies resulting from the research are to be made available to the public at large if proven to be effective and safe • New products and technologies resulting from the research are to be made available to the public sector of low- and middle-income countries • Royalties can be received through collaborations from which products or new technologies, if proven to be effective and safe, are produced and commercialized • Royalties are reinvested in research projects • Compliance was checked prior signature of memoranda of understanding Chapter 7 Page | 232 Key drivers 2. Dissemination of information, tools and policies • Publications can be a means to disseminate results from research • Publications do not generate research- derived evidence • Publications can be a way of sharing best practice and results from research, but do not constitute a real platform • Publications can contribute to national research-capacity strengthening by making available new evidence resulting from research • Through publications, results from research are made available to the public at large • Through the publications, HRP can ensure that information, results from research and new evidence are made available to low- and middle-income countries • HRP has the right to distribute copies free of charge • From the sale of the publications, HRP receives royalties • Compliance was checked prior to signature of memoranda of understanding 3. Management of research projects • The main goal is to coordinate or conduct research • Generate and synthesize research- derived evidence • Do not provide a platform for sharing best practices • Do not contribute directly to national research-capacity strengthening • New evidence, products, technologies, interventions and delivery approaches of global or national relevance are often the result of the collaboration • If research is successful, the ultimate goal is to made results available on a wide scale • New products and results resulting from the research are to be made available to the public sector in low- and middle-income countries • No royalties, but reimbursement of loans or restitution of the material can be required • Compliance was checked prior to signature of memoranda of understanding 4. Fundraising through commercial activities • Not in line with HRP’s main goal • Do not contribute to HRP's mandate derived from the UN Secretary-General's strategy (3) • Not applicable to this kind of collaboration • Not applicable to this kind of collaboration • Not applicable to this kind of collaboration • Royalties are received from the sale of goods built on the image of HRP’s initiatives • Commercial activities are not compliant with WHO principles, policies and guidelines HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 233 6.1 Identification of new products or technologies • Goal: to develop, or further develop, a product or new technology for use in the field. • Key elements: - HRP arranges and finances the trials and the research to prove whether the new product or technology is effective for use in the field; - the identified partner supplies the new products or technology in sufficient quantities to allow the performance of the trials or further research studies. • Role of the private sector and civil society: - the private sector mainly provides new products or technologies; - civil society supports the trials of new products and technologies in the field. 6.2 Dissemination of information, tools and policies • Goal: to disseminate information, tools and policies resulting from the research activities coordinated by HRP. • Key elements: - HRP licenses the right to publish the information or to make the new research tools and policies available; - an identified partner produces and markets any new publications. • Role of the private sector and civil society: - the private sector rarely participates in or supports the publication and dissemination of information and tools and the launch of new policies. This is due to the low commercial interest and limited potential returns stemming from these activities; - civil society supports the publication and dissemination of information, new tools and policies, as these contribute to the achievement of their own objectives and goals. Moreover, through their networks, these organizations have the capacity to reach out to large audiences of potential beneficiaries in low- and middle-income countries. 6.3 Management of research projects • Goal: to prioritize and carry out studies and research projects within the available funding. • Key elements: - HRP manages studies and research projects; - an identified partner supports the studies or the research projects by providing funds, equipment or products. • Role of the private sector and civil society: - the private sector’s role consists mainly in providing funds and equipment for the research projects; - civil society also provides funds, but contributes further with human resources and knowledge in the different research areas. 6.4 Fundraising through commercial activities • Goal: to raise additional funds in order to support reproductive health research projects and advocacy activities. HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 234 • Key elements2: - HRP develops a new initiative aiming to raise funds through commercial activities to be carried out in collaboration with the private sector; - the private sector uses the images of the initiative to produce and market products in order to raise funds; - funds are then returned to HRP and reinvested to support reproductive health research projects and advocacy activities. • Role of the private sector and civil society: - the private sector plays a key role within the fundraising mechanisms, through both production and commercialization of products; - the funds raised through these initiatives should contribute to support civil society’s research projects and activities. However, a civil society organization can also help promote these fundraising initiatives through its own networks (e.g. advertising the initiative on their web site). 7 Findings from analysis of existing collaborations on the basis of the key drivers identified Table 1 sets out the findings from the analysis and each category is commented on more fully under the headings that follow. 7.1 Collaborations focused on the identification of new products or technologies These collaborations: • are aligned with HRP’s key goal of promoting, coordinating, supporting, conducting and evaluating research in human reproduction. They aim mainly to conduct research with particular attention to the needs of low- and middle-income countries; • partially contribute towards the Secretary-General’s Global strategy for women’s and children’s health (3), as they generate and synthesize research-derived evidence. However, they do not provide any platform for sharing best practices, evidence on cost-effective interventions and research findings; • do not necessarily contribute to national research-capacity strengthening. However, through the collaborations, new products and technologies are produced and can be made available in low- and middle-income countries; • promote the wide availability of new health-related products to the public, particularly in low- and middle-income countries; • ensure that products are made available to the public sector in low- and middle-income countries. This is always a condition reflected in the memoranda of understanding. However, monitoring the efforts of companies or civil society organizations to ensure products’ availability is a challenge; • give HRP the possibility to claim royalties from the sales of the products in high-income countries by the private company or civil society organization. However, it is noted that there is no consistency across different initiatives and regarding the way in which royalties are negotiated. Royalties should be proportional to the contribution made by HRP to 2 It should be noted that this type of collaboration is still at its embryonic stage and different options for its implementation are still being discussed and considered within HRP. HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 235 develop or prove the efficacy of the new product or technology. In addition, it is noted that accountability related to the payment of royalties and reinvestment of the funds received needs to be improved; • are compliant with WHO legal principles, policies and guidelines, as the WHO legal office was involved since the beginning of the discussion for the establishment of the collaborations. In addition: • the identification of new products or technologies relies on both the private sector and civil society in a complementary manner, since the private sector focuses on the provision of new products and technologies and civil society organizations support the trials carried out by HRP in the field. 7.2 Collaborations aiming to disseminate information, tools and policies These collaborations: • are aligned with HRP’s key goal, as publications of information and tools are means to share and disseminate results from research, in both low- and middle income countries and high- income countries; • do not generate research-derived evidence, but constitute the channels to share the results of the research. They become a platform or a support for sharing best practices, evidence or cost-effective interventions; • can be a way to reinforce the national research capacity of low- and middle-income countries, by making information and tools, resulting from research, available to the public at large; • give the possibility to disseminate information and tools in low- and middle-income countries, as HRP reserves the right to distribute a number of copies of the different publications free of charge (to be agreed on a case-by-case basis); • generate royalties from the sale of the publications; these can be then reinvested in new research projects; • are negotiated with the support of the legal office, which is involved from the beginning of the discussions with the identified partner and in order to advise HRP. In addition: • the participation of the private sector in the dissemination of information, tools and policies is limited due to the lack of commercial interest pertaining to these activities. Conversely, these collaborations are paramount to the civil society, as they support the activities carried out in the field and facilitate their work and the achievement of their own goals. 7.3 Collaborations based on the management of research projects These collaborations: • are aligned with HRP’s key goal, as the main aim of loans and donations is to support research projects; • contribute towards the UN Secretary-General’s Global strategy for women’s and children’s health (3), although they do not establish a platform for sharing the results; HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 236 • do not directly contribute to national research-capacity strengthening, unless this is set up beforehand as a specific goal of the collaboration. However, the results of the research are then made available at national and global level by HRP; • have the ultimate goal to make the results available on a wide scale, if results from the research are successful; • make results of research available to low- and middle-income countries. However, as these collaborations are mainly focused on the research phase, the means to make the results available often need to be discussed at a second stage in the collaboration; • do not generate royalties or revenues, as they mainly focus on research activities. However, in the case of loans, the equipment or amounts are to be given back or reimbursed by HRP. In the case of products being made available to HRP, the quantities not used for the research or studies are to be returned to the supplier; • are negotiated in close collaboration with the WHO legal office, which is involved in the negotiations from the beginning of the discussions to ensure compliance with WHO policies and guidelines, and to carry out due conflict of interest checks. In addition: • civil society organizations are key players within these collaborations, as they provide funds, human resources and knowledge to carry out research. This is essential to implement research projects, as HRP itself has limited capacity. The civil society also benefits from these collaborations with HRP, since the latter acts as a door opener with the ministries of health and therefore facilitates the conduct of research-related activities within countries. 7.4 Collaborations aiming to carry out fundraising through commercial activities These collaborations: • are not in line with HRP’s main goal to promote, coordinate, support, evaluate or conduct research. However, funds to be raised through these activities can potentially be allocated to support and fund new research projects; • do not directly contribute towards HRP’s mandate deriving from the UN Secretary- General’s Global strategy for women’s and children’s health (3). However, they contribute to one of the key drivers of this new call for action, which consists of the identification of innovative mechanisms to raise funds to support initiatives related to women’s and children’s health; • do not contribute to the reinforcement of national research capacity. However, funds to be raised can potentially be allocated to activities aiming so to do; • do not aim to ensure wide availability of new products/technologies proven to be effective, and to make them available to the public-sector agencies in low- and middle-income countries. However, this driver can constitute one of the criteria for the use of funds raised through these activities; • aim to establish licensing agreements for use of the image of HRP’s initiatives for the production and commercialization of products. From the sales these products, HRP receives a percentage in royalties; • are not totally compliant with WHO principles, policies and guidelines, as the aim to carry out commercial activities to raise funds goes beyond HRP’s and WHO’s core mandate. HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 237 In addition: • commercial activities (from the production to the commercialization of products) go beyond the scope of work, the legal mandate and the actual capacity of HRP (in particular in terms of human resources). Therefore, collaboration with the private sector is a key for these initiatives that aim to raise additional funds; • HRP’s aim is to use the funds raised through commercial activities carried out in collaboration with the private sector, to finance research projects or other related activities implemented jointly with civil society organizations. 8 Conclusions and recommendations Drawing from the findings presented above, the conclusions and recommendations have been regrouped in three main areas where room for improvement has been identified. 8.1 Engagement with the private sector and civil society 8.1.1 Conclusions • HRP has a key role to play in supporting the implementation of the UN Secretary-General's Global strategy for women’s and children’s health (3), particularly in order to generate and synthesize research-derived evidence, and to provide a platform for sharing best practices, evidence on cost-effective interventions and research findings. • HRP has engaged in a number of collaborations with the private sector and civil society over the years, and has hence been able to leverage the strengths and expertise of these sectors to further its objectives. However, these collaborations have taken place in an ad hoc manner and on opportunistic basis. Moreover, teams within HRP have not tended to coordinate its activities when collaborating with the same civil society organization, particularly with international NGOs that work with different HRP teams. • No clear vision or specific strategy has been put forward to engage with those sectors that are active or willing to be active in the field of reproductive health. Although the collaborations have overall been in line with HRP’s key goal of enhancing research in human reproduction, some cases were identified where the primary aim was to raise additional funds, instead of support research, thus only indirectly contributing to HRP’s main goal. • In addition, the two objectives of providing a platform for sharing best practices, evidence on cost-effective interventions and research findings, and contributing to the strengthening of national research capacity have not been part of any of the existing collaborations. • Some collaborations with civil society organizations were set up by HRP’s teams on an informal basis without a legal framework formalizing the commitments of the parties involved. In some instances, this has led to unclear roles and responsibilities and lack of accountability, which could challenge successful collaborations. 8.1.2 Recommendations • HRP needs to develop a clear vision, accompanied by a detailed strategic plan to realize it, to strengthen the engagement with the private sector and civil society. This would allow HRP to better leverage its high-level expertise and unique position within the UN system, to maximize the benefits of these collaborations. • This vision and strategy should stipulate HRP-specific goals to engage with the focus sectors. These goals and their relationship with one another must be determined and HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 238 validated by HRP, with the assistance of process and subject experts if needed. Moreover, once agreed and defined, objectives can become part of the HRP strategy. • HRP can also consider the possibility of agreeing on a set of criteria for selection of the type of collaborations that would best serve its needs, and in order to assess new opportunities. A more structured process would increase transparency and accountability. It is noted that the definition of the criteria will require further consultation within HRP and its key stakeholders. • In order to become a key platform for sharing best practices, evidence and cost-effective interventions, HRP should strive to develop and strengthen a sense of community among all stakeholders dedicated to research on human reproduction. Consequently, HRP should consider arranging a global annual meeting, also involving the private sector and civil society. However, the platform should also be supported by other tools, such as newsletters, e-mails, social media, webinars, workshops, etc. These tools would help leverage further HRP’s relations with its stakeholders and identify new opportunities for collaboration. Moreover, this event would contribute to raising HRP’s profile and leadership by giving the opportunity to communicate clear messages regarding its activities and achievements. • To promote and ensure easy and wide access to best practices, HRP should consider further developing the existing website to make it a live web platform including e-discussion facilities and complemented by social media tools. These should be regularly updated and further promoted to ensure they keep a high level of relevance across the target audience. • Regarding civil society, and in particular the international NGOs, HRP should also consider the possibility of appointing a person to act as a focal point, who would work closely with the Programme director in order to facilitate the relations with these key organizations. This person would be responsible for further leveraging existing relations at a programme level and facilitating new collaborations at the level of teams. • Although the areas of work change from team to team within HRP, the collaboration with international NGOs could be further strengthened by applying a more coordinated approach at a programme level and by promoting further exchange across teams. • HRP should systematically adopt legal agreements for collaboration with civil society organizations. This would ensure roles and responsibilities are clearly defined and accountability is strengthened. • In the instances where HRP has a number of different collaborations with the same civil society organization, it could consider the adoption of a framework agreement between the separate institutions, and under which different teams and their respective counterparts would be able to collaborate in a flexible manner, but with the required legal coverage. 8.2 Regular monitoring of engagement with the private sector and civil society 8.2.1 Conclusions • No monitoring and evaluation framework has been put in place to assess the extent to which each of the collaborations has met the objectives. Nor is there any post–ante evaluation of the Programme’s achievement and value for money provided. • Some difficulties were experienced in gaining access to updated financial information regarding the collaborations identified and reviewed, in particular regarding the collection HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 239 of royalties. Therefore, it was not possible to assess value for money of the different collaborations. 8.2.2 Recommendations • HRP should consider the possibility of adopting a performance framework when engaging with the private sector and civil society. In this framework, defined strategic goals should be the pillars of the collaborations and become part of the overall HRP strategy. • The strategic goals should then be translated into more concrete objectives for each team and included into their workplans. Indicators and targets should be determined up front for each objective (outcomes and outputs) and included in each team’s performance framework, in order to monitor progress and results. • Building further on the teams’ workplans, the HRP manager should adopt a dashboard to capture, summarize and report on the progress made by each team on a regular basis. Moreover, this would be a tool to ensure alignment of all team to the HRP overall strategy and to contribute to the realization of the strategic objectives. • Use of a logical framework approach is advised, whereby objectives are placed in a hierarchical order. This approach needs to include definition of: - the expected impact of these collaborations overall; - the intended outcomes of each category of collaboration; - the specific outputs and inputs related to each collaboration. • HRP should start immediately gathering all financial and nonfinancial information regarding the collaborations with the private sector and civil society. This can be done by setting up a database for HRP management to conduct regular monitoring and have easy access to the information. 8.3 Structures for engaging with the private sector and civil society 8.3.1 Conclusions • HRP is currently raising funds through the royalties received on the basis on the commercialization of new products or technologies resulting from the research projects conducted in collaboration with the private sector or civil society. However, HRP is considering raising additional funds through new innovative mechanisms, particularly in collaboration with the private sector, which would entail the production and marketing of goods featuring images of HRP initiatives. However, these types of collaborations go beyond the core mandate of HRP and WHO, and may thus not be eligible within the current HRP structure. 8.3.2 Recommendations • When HRP has defined the key objectives related to the engagement with the private sector and civil society, it needs to assess the adequacy of the current structure to meet those objectives. • Alternative options regarding the structure may be explored through a feasibility study, including an analysis of the specific advantages and disadvantages and related cost implications (value for money) of each potential structure to be adopted. • For research projects, HRP should continue directly to engage with the private sector and civil society, through its current mandate and executing agency (WHO). HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 240 • When products or technologies resulting from the research projects are proven to be effective for use in the field, and can be commercialized by a private sector collaborating party, HRP should make sure that royalties are properly negotiated and that funds then received are properly monitored and reinvested. • If the collaborating party is a civil society organization entering into commercial agreements with the private sector for the production and commercialization of a new product or technology, HRP needs to ensure that the collaborative agreement (i) provides for public sector pricing to ensure availability of new products or technology in low- and middle- income countries, and (ii) receives a fair compensation, based on the investment made for development of the new product or technology. 9 The way forward Recommendations for the way forward to strengthen collaboration with the private sector and civil society are presented under the headings that follow. 9.1 Engagement with the private sector and civil society • HRP should develop a concept paper for discussion, outlining the vision for strengthening the engagement with the private sector and civil society and describing the related strategic goals. The paper should be further presented to PCC, in order to agree on the next steps. • Following the consultation with PCC, HRP should consider organizing a workshop to further develop, discuss and validate its vision for engaging with the private sector and civil society, and the related strategic goals. This workshop could also be the opportunity for HRP teams to discuss and set specific objectives that should contribute towards achieving the strategic goals. • HRP’s management should further discuss the possibility of convening an annual global meeting on reproductive health. Discussion should aim to set the objectives and expected outcomes of the initiative, together with its teams and other key stakeholders. • HRP’s management should appoint an individual as a focal point, who would work closely with the director in order to coordinate the above and facilitate relations with all stakeholders, including also the private sector and civil society. • HRP should start discussing with the WHO legal office the opportunity to adopt framework agreements with those organizations from civil society that collaborate with HRP on a regular basis, and with different HRP teams. 9.2 Regular monitoring of engagement with the private sector and civil society • Once a clear vision and strategic goals have been defined, and specific objectives agreed for each team, HRP should work on the development of a monitoring and evaluation framework, in order to keep track and assess performance on a regular basis and against a series of defined indicators. The indications would need to be agreed on the basis of the specific objectives set by each team. A dashboard could be then developed as a tool for HRP’s management to follow up on progress made by the different teams. • In parallel to the above, HRP should create a database, in order to gather all the financial and non-financial information regarding the collaborations with the private sector and civil society. HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 241 9.3 Structures for engaging with the private sector and civil society • HRP should conduct a study to identify the optimal structure to launch new innovative mechanisms and initiatives that aim to raise funds through commercial activities. It is to be noted that the study would identify all potential structures and review them against a series of criteria developed on the basis of the specific needs of HRP, in order to select the one that is best suited. Moreover, the study would highlight the opportunities and challenges of each structure. The options to be assessed would include, but not be limited to: - partner with an existing foundation; - create an independent Swiss foundation; - Set up an NGO; - outsource the initiatives to the private sector. Irrespective of the operational cycle of its governing body (PCC), HRP should take immediate action to start discussing the above recommendations, in order agree on the next steps and prepare a proposal. The proposal will then need to be submitted to PCC for review and approval. HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 242 References 1. Memorandum on the administrative structure of the Special Programme of Research, Development and Research Training in Human Reproduction (HRP). Geneva, World Health Organization, 1998 (http://www.who.int/hrp/governance/HRP_MoU_Admin_Structure_rev_2012.pdf, accessed 28 January 2013). 2. UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction. HRP programme budget for 2012–2013 (including medium-term strategic plan for 2012–2015). Geneva, World Health Organization, 2011 (http://www.who.int/reproductivehealth/publications/reports/rhr_hrp_11_09/en/index. html, accessed 28 January 2013) (WHO/RHR/HRP/11.09). 3. United Nations Secretary-General. Global strategy for women’s and children’s health. New York, United Nations, 2010 (http://www.who.int/pmnch/topics/maternal/201009_globalstrategy_wch/en/index.html , accessed 28 January 2013). 4. Guidelines on working with the private sector to achieve health outcomes. Geneva, World Health Organization, 2000 (http://apps.who.int/gb/archive/pdf_files/EB107/ee20.pdf, accessed 28 January 2013). 5. WHO policy framework for engaging and working with the commercial private sector. Geneva, World Health Organization, 2010. 6. Organization-wide expected results and indicators 4.2. In: UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction. HRP programme budget for 2012–2013 (including medium-term strategic plan for 2012–2015). Geneva, World Health Organization, 2011:38. HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 243 Appendix 1: PwC approach and methodology Approach Phase 1: kick off – agree on the terms of reference and scope of work • The team met with HRP’s key representatives in Geneva to discuss, clarify and agree on the scope of the study and to ensure a common understanding of the terms of reference. • With the support of HRP staff, HRP’s existing collaborations with the private sector and civil society were identified and mapped. Moreover, HRP staff facilitated the arrangement of the interviews with key stakeholder representatives (from both the private sector and civil society). Phase 2: perform – data gathering • With the support of HRP staff, key data and documentation regarding the existing collaborations with the private sector and civil society were gathered. The material collected consisted mainly of memoranda of understanding signed between HRP and the other interested parties for setting up the collaboration, and setting out the scope, objectives and activities for each collaboration. • Against this background, their key representatives were interviewed, in order to further understand the triggers and motivations for both the private sector and civil society partners to engage in the collaboration with HRP. Phase 3: deliver – data analysis and production of the report • The team consolidated the material and reviewed it, drawing out the key findings in order to define specific conclusions and actionable recommendations. • As required, the team gathered additional information or clarifications through follow-up phone calls and interviews. Methodology Enhance understanding of HRP from a governance perspective • The team examined the HRP core document, including its Memorandum of Understanding with WHO (1) and the views expressed during stakeholder interviews, to understand better HRP’s current structure and functioning, and its role within the overall structure of WHO. • The team focused in particular on HRP’s current legal status, in order to identify potential limitations and room for improvement from a legal perspective. Identify key HRP drivers for engaging in collaborations with the private sector and civil society • A series of key drivers related to HRP’s core mission and activities were identified. • This information was complemented with the inputs from HRP staff. • The main objective of this exercise was to identify the main incentives for HRP to develop collaborations with both the private sector and civil society, and to understand the limitations that may constrain these partnerships. HRP External Evaluation 2008−2012 (DRAFT) Chapter 7 Page | 244 • It is noted that the proposed drivers represent a set of principles that may be further developed by HRP on the basis of additional consultations. Review the existing collaborations between HRP and the private sector and civil society • The team reviewed the information collected, notably in the memoranda of understanding regulating each collaboration, and conducted further interviews to understand in more depth the nature, and identify the key elements, of these collaborations. • For reasons of confidentiality regarding each of the collaborations, details are not presented in this report. However, later in the report, the collaborations are regrouped into four main categories, based on their shared objectives, thematic areas and role. • It is also noted that the categories have been defined for the sole purpose of this study and are not exhaustive. Analyse existing collaborations on the basis of the key drivers identified • The extent to which each collaboration matched each driver was analysed and the main findings were identified. Draw conclusions and propose recommendations for the way forward • Drawing on the above, conclusions were drawn as a basis on which to propose recommendations for the way forward, aiming to strengthening HRP’s engagement with the private sector and civil society. Page | 245 Chapter 8 Bibliometric analysis Karen Gurney Manager, Bibliometric Reporting, Thomson Reuters (Evidence), Leeds, United Kingdom of Great Britain and Northern Ireland Bethan Sheridan-Jones Data Analyst, Thomson Reuters (Evidence), Leeds, United Kingdom of Great Britain and Northern Ireland Simon Thomson Data Analyst, Thomson Reuters (Evidence), Leeds, United Kingdom of Great Britain and Northern Ireland Page | 246 Page | 247 Contents Abstract . 249 1 Introduction . 251 1.1 Special Programme of Research, Development and Research Training in Human Reproduction . 251 1.2 Objectives of the report . 251 1.3 About Thomson Reuters . 251 2 Methodology . 252 2.1 Bibliometric data and citation analysis. 252 2.2 Data source . 253 2.3 Definitions used in this report . 253 2.4 Interpretation of data and analyses . 254 3 Baseline bibliometric analysis of HRP research publications . 255 3.1 Key findings . 255 3.2 HRP publication dataset for bibliometric analyses . 256 3.3 In which journals do HRP researchers publish most frequently? . 257 3.4 Is research published by HRP well cited? . 259 3.5 Trends in the numbers of papers published by HRP researchers . 261 3.6 Trends in the citation impact of papers published by HRP researchers. 262 4 International authorship and collaboration . 264 4.1 Key findings . 264 4.2 The principal countries publishing HRP research papers . 264 4.3 Collaboration between countries publishing HRP research papers . 266 4.4 HRP research papers published by researchers based in low-and middle-income countries . 268 References . 271 Annex 1: Bibliometrics and citation analysis . 272 Annex 2: HRP research portfolio . 279 Page | 248 Abbreviations HRP UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction ISI Institute for Scientific Information mNCI mean normalized citation impact NCI normalized citation impact NCIF field-normalized citation impact PMID unique publication identifier assigned by PubMed UK United Kingdom of Great Britain and Northern Ireland UN United Nations UNDP United Nations Development Programme UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USA United States of America UT unique publication identifier (unique tag) assigned by the Web of Science WHO World Health Organization HRP External Evaluation 2008−2012 Chapter 8 Page | 249 Abstract The United Nations Development Programme (UNDP)/United Nations Population Fund (UNFPA)/United Nations Children’s Fund (UNICEF)/World Health Organization (WHO)/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP) database contained 3459 records (the “research portfolio”), of which 1903 were linked to Thomson Reuters’ citation databases (the “publication dataset”). Those records that did not match the citation databases could not be included in citation analyses in this report. The principal reasons for this are: • some niche regional journals and specialist journals in obstetrics and gynaecology are not currently abstracted by the Web of Knowledge; • some document types, such as reports are not abstracted by the Web of Knowledge. Consequently, this report has evaluated the research performance of between one half and two thirds of the HRP research portfolio and focuses on the research published in journals. It therefore captures only a specific part of the HRP’s research output over the period. • Over the 21-year time period 1990–2011 encompassed by this report, HRP researchers have authored 1842 unique publications abstracted by the Web of Knowledge. Of these, 1645 are papers (substantive journal articles and reviews). • The top 20 journals in which HRP researchers have published most frequently include titles focused on contraception, andrology and obstetrics and gynaecology, as well as substantial output in elite medical journals. • The journal category of Obstetrics & Gynaecology accounts for the highest share of HRP research, with more than one third (40.9%) of HRP papers falling into this category. This research is well cited, with a citation impact approaching twice the world average. • HRP has published fewer papers in the journal categories of General & Internal Medicine and Oncology but this research is very well cited, with a citation impact of between two and three times the world average. • While the average number of papers produced by HRP researchers per annum has fallen between the time period 1990–2007 and 2008–2011, the normalized citation impact of HRP papers has risen from 1.42 between 1990 and 2007 to 2.14 by 2008– 2011, as has the percentage of these papers that are highly cited (from 16.5% to 18.3%). These indicators of research performance are significantly above the world average, and indicative of research that is of strong international standing. This report contains detailed country-level analyses of HRP research production. As WHO is based in Geneva, Switzerland, and because over 95% of Swiss papers are authored by WHO authors, the designation “WHO/Switzerland” has been used rather than the country label Switzerland to reflect this. Overall, Australia, China, the United Kingdom of Great Britain and Northern Ireland (UK), the United States of America (USA) and WHO/Switzerland have produced the highest volumes of HRP research papers over the period 1990–2011. Comparing the most recent 4 years of the analyses (2008–2011) with the longer-term period 1990–2007, the relative percentage of research papers produced by country has increased in WHO/Switzerland, the USA, Argentina, South Africa and Brazil. By contrast, the relative percentage of research papers in China, Australia and India has decreased. Page | 250 Collaboration analysis has shown that research associated with HRP has become more collaborative in recent years. The most noticeable change is the involvement of low-income countries. In the period 1990–2007, there were no collaborative papers published with low- income countries. In the most recent 4-year period, five countries: Bangladesh, Burkina Faso, Nepal, Uganda and Viet Nam have all published at least four collaborative papers associated with HRP. There has also been increased participation in HRP by lower-middle-income countries. HRP External Evaluation 2008−2012 Chapter 8 Page | 251 1 Introduction 1.1 Special Programme of Research, Development and Research Training in Human Reproduction The Special Programme of Research, Development and Research Training in Human Reproduction (HRP) is a special United Nations (UN) programme, cosponsored by the UN Development Programme (UNDP), UN Population Fund (UNFPA), United Nations Children’s Fund (UNICEF), World Health Organization (WHO) and World Bank, and founded in 1972. It is the main instrument within the UN for research in human reproduction. The Programme is hosted by WHO, but is governed independently. Since 1990, it has trained more than 70 000 staff at regional and country centres, and has administered more than 4860 research-capacity- strengthening grants. It is currently active in seven programme areas: • promoting family planning; • improving maternal and perinatal health; • preventing unsafe abortion; • controlling sexually transmitted and reproductive tract infections; • adolescent reproductive health and sexuality; • gender issues and reproductive rights in reproductive health; • sexual health. 1.2 Objectives of the report HRP wishes to use bibliometric data as an input into its 5-yearly external review process. This report provides a baseline bibliometric analysis of research funded by HRP since 1990. This project covered: • matching the publications recorded in the HRP database to Thomson Reuters’ citation databases (see Annex 2); • identifying additional HRP publications (currently not in the HRP database) and including these in the analyses (see Annex 2); • adding these publications, once verified, to the HRP database; • baseline bibliometric analyses of HRP research, looking at trends in research output and the citation impact of HRP research over the time frame 1990–2011, with particular focus on the most recent 4-year period, 2008–2011 (see Section 3); • author analysis to show the countries in which the authors of HRP research are based, and how this pattern of authorship by country has changed over time (see Section 4). 1.3 About Thomson Reuters Thomson Reuters is the world’s leading source of intelligent information for business and professionals. It combines industry expertise with innovative technology to deliver critical information to leading decision-makers in the financial, legal, tax and accounting, healthcare, science and media markets, powered by the world’s most trusted news organisation. More information is available on their web page (http://thomsonreuters.com). Thomson Reuters Research Analytics is a suite of products, services and tools that provide comprehensive research analysis, evaluation and management. For over half a century, it has Page | 252 pioneered the world of citation indexing and analysis, helping to connect scientific and scholarly thought around the world. Today, academic and research institutions, governments, not-for-profit organizations, funding agencies and all others with a stake in research need reliable, objective methods for managing and measuring performance. More information is available on their web page (http://researchanalytics.thomsonreuters.com). Thomson Reuters Custom Analytics and Engineered Solutions (including the Evidence business of Thomson Reuters) provide reporting and consultancy services within Research Analytics, using customized analyses to bring together several indicators of research performance in such a way as to enable customers to rapidly make sense of and interpret a wide range of data points to facilitate research strategy decision-making. Their consultants have up to 20 years’ experience in research performance analysis and interpretation. They have extensive experience with databases on research inputs, activity and outputs and have developed innovative analytical approaches for benchmarking, interpretation and visualization of international, national and institutional research impact. 2 Methodology 2.1 Bibliometric data and citation analysis Research evaluation is increasingly making wider use of bibliometric data and analyses. Bibliometrics is the analysis of data derived from publications and their citations. Publication of research outcomes is an integral part of the research process and is a universal activity. Consequently, bibliometric data have a currency across subjects, time and location that is found in few other sources of research-relevant data. The use of bibliometric analysis, allied to informed review by experts, increases the objectivity of and confidence in evaluation. Research publications accumulate citation counts when they are referred to by more recent publications. Citations to prior work are a normal part of publication, and reflect the value placed on a work by later researchers. Some papers get cited frequently and many remain uncited. Highly cited work is recognized as having a greater impact and Thomson Reuters (Evidence) has shown that high citation rates are correlated with other qualitative evaluations of research performance, such as peer review (1). This relationship holds across most science and technology areas and, to a limited extent, in social sciences and even in some humanities subjects. Indicators derived from publication and citation data should always be used with caution. Some fields publish at faster rates than others and citation rates also vary. Citation counts must be carefully normalized to account for such variations by field. Because citation counts naturally grow over time, it is essential to account for growth by year. Normalization is usually done by reference to the relevant global average for the field and for the year of publication. Bibliometric indicators have been found to be more informative for core natural sciences, especially for basic science, than they are for applied and professional areas and for social sciences. In professional areas, the range of publication modes used by leading researchers is likely to be diverse, as they target a diverse, non-academic audience. In social sciences, there is also a diversity of publication modes, and citation rates are typically much lower than in natural sciences. Bibliometrics work best with large data samples. As data are disaggregated, so the relationship weakens. Average indicator values (e.g. of normalized citation impact) for small numbers of publications can be skewed by single outlier values. At a finer scale, when analysing the specific outcome for individual departments, the statistical relationship is rarely a sufficient guide by itself. For this reason, bibliometrics are best used in support of, but not as a substitute for, expert decision processes. Well-founded analyses can enable conclusions to be HRP External Evaluation 2008−2012 Chapter 8 Page | 253 reached more rapidly and with greater certainty, and are therefore an aid to management and to increased confidence among stakeholders, but they cannot substitute for review by well- informed and experienced peers. Annex 1 of this report provides the standard methodology and data definitions used in bibliometric and citation analyses. 2.2 Data source For this report, bibliometric data are sourced from Thomson Reuters’ databases underlying the Web of Knowledge, which gives access to conference proceedings, patents, websites, and chemical structures, compounds and reactions, in addition to journals. It has a unified structure that integrates all data and search terms together and therefore provides a level of comparability not found in other databases. It is widely acknowledged to be the world’s leading source of citation and bibliometric data. The Web of Science is part of the Web of Knowledge, and focuses on research published in journals and conferences in science, medicine, arts, humanities and social sciences. The authoritative, multidisciplinary content covers over 12 000 of the highest-impact journals worldwide, including open-access journals and over 150 000 conference proceedings. Coverage is both current and retrospective in the sciences, social sciences, arts and humanities, in some cases back to 1900. Within the research community, these data are often still referred to by the acronym “ISI” (Institute for Scientific Information – now the IP and Science business of Thomson Reuters). Thomson Reuters (Evidence) has extensive experience with databases on research inputs, activity and outputs and has developed innovative analytical approaches for benchmarking and interpreting international, national and institutional research impact. Granularity of analysis is an important issue. Unduly fine analysis at the level of research groups provides little comparability or connectedness, while coarse analysis may miss spikes of excellence in key areas. Journals are mapped to one or more subject categories, and every article within that journal is subsequently assigned to that category. Thomson Reuters (Evidence) uses these categories as the basis for bibliometric analysis because they are well established and informed by extensive work with the research community since inception. Papers from prestigious, “multidisciplinary” and general “biomedical” journals such as Nature, Science, BMJ, The Lancet, The New England Journal of Medicine and Proceedings of the National Academy of Sciences of the United States of America are assigned to specific categories based on the journal categories of the citing and cited references in each article. Further information about the journals included in the citation databases and how they are selected is available at http://scientific.thomsonreuters.com/mjl/. The citation analyses presented in this report will not cover conference proceedings, meeting abstracts, books, chapters in books or grey literature such as reports. It therefore captures only a specific part of the total output of the specified institutional research base(s) over the period, but this part is usually recognized as describing the most direct contribution to the research base. 2.3 Definitions used in this report HRP research portfolio: this refers to the entire body of research, including internal WHO and external reports, as well as journal articles and conference papers recorded in the HRP publications database. Not all of these outputs will be included in the citation analyses in this report. HRP publications dataset: this refers to those publications from the HRP research portfolio that have been matched to Thomson Reuters’ citation databases. Page | 254 Papers/publications: Thomson Reuters abstracts publications, including editorials, meeting abstracts and book reviews, as well as research journal articles. The terms “paper” and “publication” are often used interchangeably to refer to printed and electronic outputs of many types. In this report, the term “paper” has been used exclusively to refer to substantive journal articles, reviews and some proceedings papers, and excludes editorials, meeting abstracts or other types of publication. Papers are the subset of publications for which citation data are most informative and that are used in calculations of citation impact. Citations: the citation count is the number of times that a citation has been recorded for a given publication since it was published. Not all citations are necessarily recorded, since not all publications are indexed. However, the material indexed by Thomson Reuters is estimated to attract about 95% of global citations. Citation impact: “citations per paper” is an index of academic or research impact (as compared with economic or social impact). It is calculated by dividing the sum of citations by the total number of papers in any given dataset (so, for a single paper, raw impact is the same as its citation count). Impact can be calculated for papers within a specific research field such as clinical neurology, or for a specific institution or group of institutions, or a specific country. The citation count declines in the most recent years of any time period, as papers have had less time to accumulate citations (papers published in 2007 will typically have more citations than papers published in 2010). Field-normalized citation impact (NCIF): citation rates vary between research fields and with time; consequently, analyses must take both the field and year into account. In addition, the type of publication will influence the citation count. For this reason, only citation counts of papers (as defined above) are used in calculations of citation impact. The standard normalization factor is the world average citations per paper for the year and journal category in which the paper was published. This normalization is also referred to as “rebasing” the citation count. Mean-normalized citation impact (mNCI): the mNCI indicator for any specific dataset is calculated as the mean of the NCIF of all papers within that dataset. Highly cited (top decile) papers: using the calculated percentile data based on the ranking of each paper within the journal category, “highly cited” papers can be defined. This refers to the world’s top 10% of most frequently cited papers, taking into account the year of publication and field. Research field: standard bibliometric methodology uses the journal category as a proxy for research field. Journals are assigned to one or more categories, and every article within that journal is subsequently assigned to that category. Papers from prestigious, “multidisciplinary” and general medical journals such as Nature, Science, The Lancet, BMJ, The New England Journal of Medicine and Proceedings of the National Academy of Sciences of the United States of America are assigned to specific categories based on the journal categories of the references cited in the article. The selection procedures for the journals included in the citation databases are documented at http://scientific.thomsonreuters.com/mjl/. For this evaluation, the standard classification of Web of Science journal categories has been used. 2.4 Interpretation of data and analyses Papers: the minimum number of papers suitable as a sample for quantitative research evaluation is a subject of widespread discussion. Larger samples are always more reliable, but a very high minimum may defeat the scope and specificity of analysis. Experience has indicated that a threshold between 20 and 50 papers can generally be deemed appropriate. For work that is likely to be published with little contextual information, the upper boundary (≥50) is a desirable starting point. For work that will be used primarily by an expert, in-house group, then HRP External Evaluation 2008−2012 Chapter 8 Page | 255 the lower boundary (≥20) may be approached. Because comparisons for in-house evaluation often involve smaller, more specific research groups (compared to broad institutional comparisons), a high volume threshold is self-defeating. Smaller samples may be used but outcomes must be interpreted with caution and expert review should draw on multiple information sources before reaching any conclusions. Mean field-normalized citation impact: NCIF values for individual papers vary widely and it is more useful to consider the mNCI. This average can be at several granularities: field (either journal category or field), annual and overall (total output under consideration). When considering such mNCI data points, care must be taken to understand that these data are highly skewed and the average can be driven by a single, highly cited paper (this would be highlighted in accompanying text though not apparent from tables and figures). The world average is 1.0, so any NCIF value higher than this indicates a paper, or set of papers, that is/are cited more than average for similar research worldwide. For the purposes of research management, experience suggests that NCIF values between 1.0 and 2.0 should be considered to be indicative of research that is influential at a national level, while that cited more than twice the world average has international recognition. Table 1 summarizes the thresholds used for the various indicators discussed. Table 1 Indicator values and thresholds Indicator Threshold Number of publications (all output types) No threshold Number of papers (articles and reviews) Citation analyses based on fewer than 20 papers at any particular aggregation, e.g. year or field are not reliable. Percentage of highly cited papers (those ranked in the top decile of world papers relative to field/journal category and year) A value of more than 10% indicates better than world average. Mean normalized citation impact (mNCI – an indication of paper quality within the field) A value of more than 1.0 indicates better than the world average. 3 Baseline bibliometric analysis of HRP research publications 3.1 Key findings • The HRP research publications used in these analyses comprise 1842 publications, of which 1645 are substantive journal articles and reviews (papers) matched to citation data (see Annex 2 for details of matching the HRP database to Thomson Reuters’ citation databases). • The list of the top 20 journals in which HRP researchers have published most frequently includes journals focused on contraception, andrology and obstetrics and gynaecology, as well as elite medical journals. • The journal category of Obstetrics & Gynaecology accounts for the highest share of HRP research, with more than one third (40.9%) of HRP papers falling into this category. This research is also well cited, with a citation impact approaching twice the world average (1.71). • While the average number of papers produced per annum has fallen between the time periods 1990–2007 and 2008–2011, the mNCI of HRP papers has risen (from 1.42 between 1990–2007 to 2.14 by 2008–2011), as has the percentage of these papers that are highly cited (from 16.5% to 18.3%). These measures of citation impact are Page | 256 significantly above the world average and indicative of research of international standing. 3.2 HRP publication dataset for bibliometric analyses The HRP publication dataset for bibliometric analyses comprises 1842 publications linked to the citation databases underlying the Web of Knowledge, of which 1645 are papers with NCI data. The citation analyses do not include research published in grey literature such as reports, nor does it include meeting abstracts or books. It also does not include research published in journals not abstracted by Thomson Reuters Web of Knowledge. It is difficult to assess what proportion of the HRP research portfolio recorded in the HRP database is included in these analyses, as the database contains duplicates and document types other than journal articles, but it is estimated to be between one half and two thirds (see Figure 1 and Annex 2). Figure 1 The relationship of the HRP publications dataset to the HRP database; UT, unique tag HRP External Evaluation 2008−2012 Chapter 8 Page | 257 Figure 2 shows the categorization and share of the HRP research publications for bibliometric analysis by document type. Figure 2 Categorization of HRP research publications by document type, 1990–2011 Data and analysis: Thomson Reuters (Evidence) Well over three quarters (89.6%) of HRP publications were substantive papers (journal articles and reviews, some proceedings papers and notes). These are the document types used in citation analyses. It should be remembered that Figure 2 is based only on the research publications matched to Thomson Reuters’ databases and not the entire HRP research portfolio as recorded in the database. 3.3 In which journals do HRP researchers publish most frequently? The 20 journals used most frequently by HRP researchers in the period 1990–2011 are listed in Table 2 (a total of 148 journal titles are used more than once). There is a total of 1023 publications in these most frequently used journals – just over half (55.5%) the number of publications in the dataset. This demonstrates the uniformity of the research publications produced by HRP researchers. The core set of journals includes journals focused on contraception, andrology and obstetrics and gynaecology, with a strong focus on the latter. There is also a substantial number of publications in elite or well-regarded general medical titles such as The Lancet and BMJ. All but four of the journals in Table 2 are ranked in the top quartile (by journal impact factor) of journals in their specific research fields.1 Journals shaded in grey are in lower quartiles, that is, they are typically less frequently cited than other journals in their field. 1 Where a particular journal is associated with more than one Web of Science journal category, the one in which it performs best has been used. Page | 258 Table 2 Journals in which HRP publications have appeared most frequently (1990–2011), top 20 ranked by number of publications Journal Number of publications Journal impact factor (2011) Contraception 294 2.724 Human Reproduction 154 4.475 The Lancet 98 38.278 Bulletin of the World Health Organization 46 4.641 Biology of Reproduction 44 4.009 Cochrane Database of Systematic Reviews 43 5.715 International Journal of Andrology 38 3.591 Fertility and Sterility 36 3.564 The Journal of Clinical Endocrinology and Metabolism 35 5.967 American Journal of Obstetrics and Gynaecology 29 3.468 International Journal of Gynaecology and Obstetrics 28 2.045 Reproductive Health Matters 24 1.371 BJOG: An International Journal of Obstetrics and Gynaecology 22 3.407 Obstetrics and Gynaecology 21 4.73 Journal of Reproductive Immunology 21 2.966 Statistics in Medicine 20 1.877 Journal of Reproduction and Fertilitya 19 n/a British Medical Journal 17 14.093 Placenta 17 3.693 Journal of Biosocial Science 17 0.98 n/a, not available. a This journal is no longer abstracted in Web of Science. The most recent HRP publication in this journal was published in 1996. Table 3 lists the 20 journals with the highest journal impact factor, used more than once by HRP researchers. These range from elite medical and multidisciplinary titles (The New England Journal of Medicine, The Lancet, JAMA: Journal of the American Medical Association) to highly regarded specialist journals (Human Reproduction Update, Cell Death and Differentiation). HRP External Evaluation 2008−2012 Chapter 8 Page | 259 Table 3 Journals in which HRP publications have appeared most frequently (1990–2011), top 20 ranked by journal impact factor (2011) Journal Number of publications Journal impact factor (2011) The New England Journal of Medicine 7 53.298 The Lancet 98 38.278 Nature 1 36.28 JAMA: Journal of the American Medical Association 3 30.026 Endocrine Reviews 1 19.929 Lancet Infectious Diseases 3 17.391 PLOS Medicine 3 16.269 BMJ 17 14.093 Journal of the National Cancer Institute 1 13.757 Journal of Cell Biology 2 10.264 Proceedings of the National Academy of Sciences of the United States of America 3 9.681 Current Biology 1 9.647 Human Reproduction Update 13 9.234 Cell Death and Differentiation 1 8.849 Canadian Medical Association Journal 1 8.217 Trends in Endocrinology and Metabolism 2 8.115 Nucleic Acids Research 2 8.026 Cancer Research 1 7.856 Clinical Cancer Research 1 7.742 Epidemiologic Reviews 1 7.583 3.4 Is research published by HRP well cited? The citation impact of research, an indicator linked to the accumulation of citations, is subject specific. Typically, papers published in areas such as biomedical research receive more citations than papers published in subjects such as engineering, even if the papers are published in the same year. All data on citation impact presented in this report are therefore normalized to the relevant world average, to allow comparison between years and fields. For example, the mNCI for HRP research in obstetrics and gynaecology is the raw citation impact (citations/paper) of HRP papers in obstetrics and gynaecology for any given year of publication, divided by the raw impact (citations/paper) for all the world’s publications in obstetrics and gynaecology journals from the same year. The overall mNCI for HRP papers is 1.53 (where the world average is 1.0) for the period 1990– 2011. This can be disaggregated by journal category to show the fields in which HRP papers are most cited (see Section 3.4.1). Section 3.6 analyses the way the citation impact of HRP papers has changed since 1990. 3.4.1 In which fields is research published by HRP well cited? Standard bibliometric methodology uses the journal category as a proxy for research field. Journals are assigned to one or more categories, and every article within that journal is subsequently assigned to that category.2 Figure 3 shows the total number of papers and mNCI 2 There are exceptions to this – see Section 2.3, Definitions used in this report. Page | 260 of research in each of the top 10 most frequently used Web of Science journal categories for HRP publications. Figure 3 Total number of papers and mean normalized citation impact in the top 10 most frequently used Web of Science journal categories for HRP publications, 1990–2011 The journal category of Obstetrics & Gynaecology accounts for the highest share of HRP research, with more than one third (40.9%) of HRP papers falling into this category. This research is also well cited, with a citation impact approaching twice the world average (1.71). HRP papers published in the journal categories of General & Internal Medicine and Oncology are extremely well cited, with a citation impact between two and three times the world average, though a smaller numbers of papers have been published in these fields (see Table 4). HRP papers in journals in the category Andrology have been cited less than the world average for similar research. Table 4 Summary of volume and citation impact in the top 10 most frequently used Web of Science journal categories Web of Science journal category Number of papers Citation impact (mNCI) Obstetrics & Gynaecology 672 1.71 Public, Environmental & Occupational Health 139 1.48 Reproductive Biology 131 1.26 Endocrinology & Metabolism 106 1.19 General & Internal Medicine 76 2.41 Andrology 54 0.89 Infectious Diseases 53 1.61 Demography 49 1.21 Oncology 39 3.09 Paediatrics 29 1.50 Overall 1654 1.53 HRP External Evaluation 2008−2012 Chapter 8 Page | 261 3.4.2 Highly cited research published by HRP Average citation impact, especially on relatively small paper numbers such as in this report, can be misleading. If, for example, there is a single paper that is very highly cited, this can “pull the average up” while in general the papers may have been cited less well. Another indicator of research performance based on citation count is the percentage of highly cited papers – defined here as those ranked in the top decile of world papers relative to the field/journal category and year. If this indicator is below 10% (the world benchmark), then this would indicate that a high average citation impact may be due to one or a few very highly cited papers and not indicative of the general trend. The overall percentage of highly cited papers published by HRP is 15.7% (where the world benchmark is 10%) for the period 1990–2011. Section 3.6.1 analyses how the citation impact of HRP papers has changed since 1990. This, taken in consideration with the above world average citation impact, indicates that the research performance of HRP papers is consistently good and above the world average. 3.5 Trends in the numbers of papers published by HRP researchers The numbers of papers published by HRP researchers sharply decreased in the years 2007 and 2008 but in 2011 recovered to a number comparable to that before these years. This trend is apparent in the HRP research portfolio and is not specific to the part of the research included in these analyses. Also, the searches to identify additional papers not recorded by HRP have not removed this apparent decline, that is, a significantly larger number of papers was not found for these years than for those either before or after. The number of papers produced by HRP researchers has decreased over the period 1990 to 2007, averaging 76.8 papers per annum. While, from 2008 to 2011, the number of papers produced by HRP researchers increased year-on-year, the average for the period (65.8) is lower than the average for the previous period (see Figure 4). Figure 4 The number of HRP papers by year, 1990–2011, in two time periods 1990–2007 and 2008–2011, with averages Data and analysis: Thomson Reuters (Evidence) Page | 262 3.6 Trends in the citation impact of papers published by HRP researchers The citation impact of HRP papers has increased since 1990, from around the world average to more than twice the world average. The improvement is more marked in the most recent 4-year period since 2007, though this is partly due to the sharp drop in citation impact for the 2008 papers. Owing to the small number of papers on which the calculations of mNCI are based, data are rather “spiky” in Figure 5. However, with the exception of 3 years when the mNCI of HRP research papers was below 1.0 (the world average), typically it is well above the world average; averaging 1.42 over the period 1990–2007, and 2.14 over the period 2008–2011. This is a significant increase between the time periods, suggesting a significant improvement in the mNCI of HRP research. Figure 5 Normalized citation impact by year, 1990–2011, in two time periods 1990–2007 and 2008– 2011, with averages (axis intercept at 1.0 = world average) Data and analysis: Thomson Reuters (Evidence) 3.6.1 Trends in highly cited research published by HRP researchers Trend analysis of highly cited papers (in the world’s 10% most frequently cited papers) shows a similar pattern to that seen for NCI (compare Figures 4 and 5). Again, there has been a steady improvement since 1990, from just over 11%; however, the data for the most recent four-year period are quite variable. The average for 2008–2011 is higher than for the pre-2007 period but has not shown the consistent upturn seen in the NCI. Figure 6 shows that the percentage of HRP papers that are highly cited has broadly increased over the period 1990–2007, averaging 16.5% overall. Between 2008 and 2011, this figure rose to 18.3%. There is a marked spike around 2006 and 2007, driven by particularly highly cited papers published in The Lancet and JAMA: the Journal of the American Medical Association. HRP External Evaluation 2008−2012 Chapter 8 Page | 263 Figure 6 Percentage of highly cited (top decile) papers by year, 1990–2011, in two time periods 1990– 2007 and 2008–2011, with averages (axis intercept at 10% = world benchmark) Data and analysis: Thomson Reuters (Evidence) 3.6.2 Summary of changes in the numbers of papers and citation impact between 1990–2007 and 2088–2011 Figure 7 summarizes the data in the previous analyses in Sections 3.5 and 3.6. It shows that, while the average number of papers produced per annum has fallen between two time periods 1990–2007 and 2008–2011, the normalized citation impact has risen, as has the percentage of HRP research that is highly cited. Furthermore, the most recent indicators of research performance are well above the world average: around twice the world average/benchmark in the most recent period (citation impact = 2.14, proportion of highly cited papers = 18.3% in 2008–2011). Previous work suggests that research cited more than twice the world average is likely to be well regarded among the international research community (see Section 2.4). Figure 7 Summary of changes in the numbers of papers and citation impact between 1990–2007 and 2088–2011 (axis intercept at 1.0 = world average) Data and analysis: Thomson Reuters (Evidence) Page | 264 4 International authorship and collaboration This section of the report shows the countries in which the authors (both directly supported and collaborators) of HRP research are based, and how this pattern of authorship by country has changed over time. 4.1 Key findings • Australia, China, the United Kingdom of Great Britain and Northern Ireland (UK), United States of America (USA) and WHO/Switzerland have published the most HRP research papers over the period 1990–2011 (see Section 4.2). • Comparing the period 2008–2011 with the long-term period 1990–2007, the relative percentage of research papers produced by country has increased in Argentina, Brazil, South Africa, the USA and WHO/Switzerland. By contrast, the relative percentage of research papers in Australia, China and India has decreased (see Section 4.2). 4.2 The principal countries publishing HRP research papers WHO/Switzerland, USA, UK, China and Australia have published the most HRP research papers over the period 1990–2011 (see Figure 8). Figure 8 Top 20 countries publishing HRP research papers, by number of papers 1990–2011 Data and analysis: Thomson Reuters (Evidence) HRP External Evaluation 2008−2012 Chapter 8 Page | 265 The maps (see Figure 9) show the number of papers, as a percentage of the HRP publication dataset, published by the principal countries, broken down by three time periods: 1990–2002, 2003–2007 and 2008–2011. Table 5 additionally shows the comparison for the long-term period 1990–2007 to the recent period 2008–2011. The arrows show where the percentage change between these two periods is ≥2.5% (↑); or <2.5 and ≥–2.5% (→); or <–2.5% (↓). These analyses show that the percentage of HRP papers published by authors in WHO/Switzerland, USA, South Africa and South America, especially Argentina and Brazil, has increased. By contrast, the percentage of HRP papers published by authors in China, Australia and India has decreased. For the other countries, the percentage of HRP papers may have increased or decreased, but these changes are marginal. Figure 9 Maps showing how the contribution to the HRP publication dataset has changed since 1990 at country level Page | 266 Table 5 Principal countries associated with publishing HRP research papers, and change over time 1990–2002 2003–2007 1990–2007 2008–2011 Number % Number % Number % Number % Change HRP papers 1067 — 315 — 1382 — 263 — — Argentina 32 3.0 18 5.7 50 3.6 28 10.6 ↑ Australia 131 12.3 25 7.9 156 11.3 16 6.1 ↓ Brazil 31 2.9 6 1.9 37 2.7 28 10.6 ↑ Canada 45 4.2 11 3.5 56 4.1 16 6.1 → Chile 67 6.3 9 2.9 76 5.5 9 3.4 → China 147 13.8 47 14.9 194 14.0 18 6.8 ↓ France 27 2.5 14 4.4 41 3.0 14 5.3 → Germany 42 3.9 8 2.5 50 3.6 6 2.3 → Hungary 29 2.7 2 0.6 31 2.2 1 0.4 → India 59 5.5 21 6.7 80 5.8 8 3.0 ↓ Indonesia 45 4.2 1 0.3 46 3.3 3 1.1 → Italy 15 1.4 8 2.5 23 1.7 9 3.4 → Kenya 24 2.2 10 3.2 34 2.5 3 1.1 → Mexico 46 4.3 6 1.9 52 3.8 6 2.3 → South Africa 14 1.3 20 6.3 34 2.5 37 14.1 ↑ Sweden 61 5.7 22 7.0 83 6.0 15 5.7 → Thailand 64 6.0 12 3.8 76 5.5 11 4.2 → UK 225 21.1 81 25.7 306 22.1 59 22.4 → USA 267 25.0 90 28.6 357 25.8 106 40.3 ↑ WHO/Switzerland 231 21.6 133 42.2 364 26.3 175 66.5 ↑ 4.3 Collaboration between countries publishing HRP research papers Section 4.2 showed that the contribution made to HRP by Africa and South America, in terms of publication volume, has increased since 1990. However, this is country specific; for example, while authors in Argentina and Brazil now publish an increased percentage of HRP papers, those in Mexico and Chile have decreased their relative contribution. The visualizations in Figures 10 and 11 illustrate the international collaboration at country level in the HRP publications dataset for the long-term period 1990–2007 and the recent period 2008–2011, using lines to link two countries that have collaborated on published research papers. Such visualizations rely on subjective thresholds and two thresholds have been used: • higher threshold: the lines link two countries that have collaborated on at least one paper per year on average for the time period; • lower threshold: the lines link two countries that have collaborated on at least one paper every 2 years on average for the time period In both figures, the countries are arranged around the cartwheel, from the top, in alphabetical order within World Bank income category: high income, lower middle income, low income and upper middle income. Figure 10 shows that, in line with increasing contribution to the HRP research papers, WHO/Switzerland, UK and USA have expanded their collaboration worldwide. HRP External Evaluation 2008−2012 Chapter 8 Page | 267 Some high-income countries, for example, Hungary and Slovenia, were active in this programme pre-2007 but no longer contribute to HRP. There has been little change in those middle-income countries participating in HRP – those who were publishing pre-2007, in general, have published most recently too. The most noticeable change is the involvement of low-income countries. In the period 1990– 2007, there were no collaborative papers published with low-income countries but in the most recent 4-year period, five countries: Bangladesh, Burkina Faso, Nepal, Uganda and Viet Nam have all published at least four collaborative papers associated with HRP. Figure 10 Collaboration cartwheels showing how research collaboration at country level has changed between 1990–2007 and 2008–2011, higher threshold = one paper per year on average over the period In Figure 11, the collaboration cartwheels again show the overall increase in collaboration centred around the high-income countries: Canada, Sweden, WHO/Switzerland, UK and USA. At this lower threshold, which will pick up more collaborations, among the high-income countries, Hungary, Israel, Saudi Arabia and Singapore have become less active, while Austria, Denmark and New Zealand now contribute to HRP. There has been more change in those middle-income countries participating in HRP – especially among the lower-middle-income group, which has expanded to include some south American and central Asian countries. As with the higher-threshold analysis, the most noticeable change is the involvement of low- income countries, which has increased from one (Kenya) to eight, including two more African countries (Ethiopia and Kenya) and Cambodia, as well as the five countries listed earlier. Page | 268 Figure 11 Collaboration cartwheels showing how research collaboration at country level has changed between 1990–2007 and 2008–2011, lower threshold = one paper every two years on average over the period 4.4 HRP research papers published by researchers based in low-and middle-income countries The analyses in this section look at the subset of HRP research papers published by researchers based in low- and middle-income countries.3 • The percentage of HRP research papers published where all the authors are from low- or middle-income countries has decreased since 1990 by almost half (from 23.2% in 1990–2002 to 13.7% in 2008–2011, see Figure 12). • Over the same period, the percentage of HRP research papers where at least one author is from a low- or middle-income country has increased by nearly 20% (from 43.4% in 1990–2002 to 63.9% in 2008–2011, see Figure 12). • WHO/Switzerland, the USA, the UK, China and Australia were the countries that collaborated most frequently on papers where the first author is affiliated with a low- or middle-income country (see Figure 13). 3 Defined by the WHO as all countries except Australia, Austria, Belgium, Canada, Croatia, Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Israel, Italy, Japan, Latvia, the Netherlands, New Zealand, Norway, Poland, Portugal, Republic of Moldova, Romania, the Russian Federation, Slovakia, Slovenia, Spain, Sweden, Switzerland, Ukraine, UK and USA. HRP External Evaluation 2008−2012 Chapter 8 Page | 269 Figure 12 Contribution of low- and middle-income countries to publication of HRP research, 1990– 2011a aData for first-author analyses are available for the period mid-2008 to the end of 2011 only (see text in Section 4.4). Figure 13 Top countries collaborating on papers where the first author is affiliated to a low- or middle- income country, by number of papers from mid-2008 to the end of 2011 Data and analysis: Thomson Reuters (Evidence) Page | 270 Since mid-2008, Thomson Reuters has included, as part of the metadata associated with the publications it indexes, the explicit links between authors and addresses. Prior to this date the authors and addresses associated with papers were extracted and indexed, but the links between them were not. Of the 260 HRP papers for which author–addresses link metadata are available, 104 (40.0%) have a first author that is affiliated with a low- or middle-income country. Around half of those HRP papers with a first author that is affiliated with a low- or middle-income country have a coauthor affiliated with a Swiss address (including the address of the WHO Headquarters). HRP External Evaluation 2008−2012 Chapter 8 Page | 271 References 1. Adams J et al, Evidence Ltd. (2002) Maintaining research excellence and volume. A report for the Higher Education Funding Councils for England, Scotland and Wales and for universities UK. Leeds, Evidence Ltd, 2002 (http://www.evidence.co.uk/downloads/maint-research-exellence.pdf, accessed 3 December 2012). 2. Garfield E. Citation indexes for science – new dimension in documentation through association of ideas. Science, 1955, 122:108–111. 3. The Organisation for Economic Co-operation and Development. Frascati manual: proposed standard practice for surveys on research and experimental development, 6th ed. Paris, OECD Publishing, 2002. 4. Yancey R. Fifty years of citation indexing and analysis. Leeds, Thomson Reuters, 2005 (http://thomsonreuters.com/products_services/science/free/essays/50_years_citation_indexin g/, accessed 18 December 2012). Page | 272 Annex 1: Bibliometrics and citation analysis Bibliometrics are about publications and their citations. The academic field emerged from “information science” and now usually refers to the methods used to study and index texts and information. Publications cite other publications. These citation links grow into networks, and their numbers are likely to be related to the significance or impact of the publication. The meaning of the publication is determined from keywords and content. Citation analysis and content analysis have therefore become a common part of bibliometric methodology. Historically, bibliometric methods were used to trace relationships among academic journal citations. Now, bibliometrics are important in indexing research performance. Bibliometric data have particular characteristics of which the user should be aware, and these are considered here. Journal papers (publications, sources) report research work. Papers refer to or “cite” earlier work relevant to the material being reported. New papers are cited in their turn. Papers that accumulate more citations are thought of as having greater “impact”, which is interpreted as significance or influence on their field. Citation counts are therefore recognized as a measure of impact, which can be used to index the excellence of the research from a particular group, institution or country. The origins of citation analysis as a tool that could be applied to research performance can be traced to the mid-1950s, when Eugene Garfield proposed the concept of citation indexing and introduced the Science Citation Index, the Social Sciences Citation Index and the Arts and Humanities Citation Index, produced by the Institute of Scientific Information (ISI – currently the IP and Science business of Thomson Reuters) (2). Citations can be counted, but they are only “indicators” of impact or quality – not metrics. Most impact indicators use average citation counts from groups of papers, because some individual papers may have unusual or misleading citation profiles. These outliers are diluted in larger samples. Data source The data used in this report come from the Thomson Reuters’ databases underlying the Web of Knowledge, which gives access not only to journals but also to conference proceedings, books, patents, websites, and chemical structures, compounds and reactions. It has a unified structure that integrates all data and search terms together and therefore provides a level of comparability not found in other databases. It is widely acknowledged to be the world’s leading source of citation and bibliometric data. The Web of Science is one part of the Web of Knowledge, and focuses on research published in journals, conferences and books in science, medicine, arts, humanities and social sciences. The Web of Science was created as an awareness and information-retrieval tool but it has acquired an important secondary use as a tool for research evaluation, using citation analysis and bibliometrics. Data coverage is both current and retrospective in the sciences, social sciences, arts and humanities, in some cases back to 1900. Within the research community this data source is often still referred to by the acronym “ISI”. Unlike other databases, the Web of Science and underlying databases are selective; that is, the journals abstracted are selected using rigorous editorial and quality criteria. The authoritative, multidisciplinary content covers over 12 000 of the highest-impact journals worldwide, including open-access journals, and over 150 000 conference proceedings. The abstracted HRP External Evaluation 2008−2012 Chapter 8 Page | 273 journals encompass the majority of significant, frequently cited scientific reports and, more importantly, an even greater proportion of the scientific research output that is cited. This selective process ensures that the citation counts remain relatively stable in given research fields and do not fluctuate unduly from year to year, which increases the usability of such data for performance evaluation. Evidence, now as part of Thomson Reuters, has extensive experience with databases on research inputs, activity and outputs, and has developed innovative analytical approaches for benchmarking and interpreting international, national and institutional research impact. Database categories The source data can be grouped in various classification systems. Most of these are based on groups of journals that have a relatively high cross-citation linkage and naturally cluster together. Custom classifications use subject maps in third-party data such as the Organisation for Economic Co-operation and Development (OECD) categories set out in the Frascati manual (3). Thomson Reuters typically uses the broader field categories in the Essential Science Indicators system and the finer journal categories in the Web of Science. There are 22 fields in Essential Science Indicators and 254 fields in the Web of Science. In either case, Thomson Reuters’ bibliometric analyses draw on the full range of data available in the underlying database, so analyses in their reports will differ slightly from anything created “on the fly” from data in the web interface. The lists of journal categories in these systems can be found at http://ip- science.thomsonreuters.com/mjl/scope/scope_sci/ and the subject areas for Essential Science Indicators can be found at http://incites- help.isiknowledge.com/incites_112_live/globalComparisonsGroup/globalComparisons/subjAre aSchemesGroup/essentialScienceIndicators.html. Most analyses start with an overall view across the data, then move to a view across broad categories and only then focus in at a finer level in the areas of greatest interest to policy, programme or organizational purpose. Assigning papers to addresses A paper is assigned to each country and each organization whose address appears at least once for any author on that paper. One paper counts once and only once for each assignment; however, many address variants occur for the country or organization. No weighting is applied. For example, a paper has five authors, thus: Author Organization Country At institutional level At country level Gurney KA University of Leeds UK Counts for University of Leeds Counts for UK Adams J University of Leeds UK No gain for University of Leeds No gain for UK Kochalko D University of California, San Diego (UCSD) USA Counts for UCSD Counts for USA Munshi S Gujarat University India Counts for Gujarat University Counts for India Pendlebury D University of Oregon USA Counts for University of Oregon No gain for USA So this one paper with five authors would be included once in the research output for each of four universities and once in the research output for each of three countries. Work carried out within Thomson Reuters, and research published elsewhere, indicates that fractional weighting based on the balance of authors by organization and country makes little difference to the conclusions of an analysis at an aggregate level. Such fractional analysis can Page | 274 introduce unforeseen errors in the attempt to create a detailed but uncertain assignment. Partitioning credit would make a greater difference at a detailed, group level but the analysis can then be manually validated. Citation counts A publication accumulates citation counts when it is referred to by more recent publications. Some papers get cited frequently and many get cited rarely or never, so the distribution of citations is highly skewed. Why are many papers never cited? Certainly some papers remain uncited because their content is of little or no impact, but that is not the only reason. It might be because they have been published in a journal not read by researchers for whom the paper might be of interest. It might be that they represent important but “negative” work reporting a blind alley to be avoided by others. The publication may be a commentary in an editorial, rather than a normal journal article and thus of general rather than research interest. Or it might be that the work is a “sleeping beauty” that has yet to be recognized for its significance. Other papers can be very highly cited: hundreds, even thousands of times. Again, there are multiple reasons for this. Most frequently cited work is being recognized for its innovative significance and impact on the research field of which it speaks. Impact here is a good reflection of quality: it is an indicator of excellence. But there are other papers that are frequently cited because their significance is slightly different: they describe key methodology; they are a thoughtful and wide-ranging review of a field; or they represent contentious views that others seek to refute. Citation analysis cannot make value judgments about why an article is uncited or about why it is highly cited. The analysis can only report the citation impact that the publication has achieved. It is normally assumed, based on many other studies linking bibliometric and peer judgments, that high citation counts correlate, on average, with the quality of the research. Figure 14 shows the skewed distribution of more or less frequently cited papers from a sample of UK-authored publications in cell biology. The skew in the distribution varies from field to field. It is to compensate for such factors that actual citation counts must be normalized, or rebased, against a world baseline. Figure 14 Distribution of citation data HRP External Evaluation 2008−2012 Chapter 8 Page | 275 Thomas Reuters does not seek to account separately for the effect of self-citation. If the citation count is significantly affected by self-citation, then the paper is likely to have been infrequently cited. This is therefore only of consequence for low-impact activity. Studies show that for large samples at national and organizational level, the self-citation has little or no effect on the analytical outcomes and would not alter interpretation of the results. Time factors Citations accumulate over time. Older papers therefore have, on average, more citations than more recent work. Figure 15 shows the pattern of citation accumulation for a set of 33 journals in the journal category Materials Science, Biomaterials. Papers less than 8 years old are, on average, still accumulating additional citations. The citation count goes on to reach a plateau for older sources. The graph shows that the percentage of papers that have never been cited drops over about 5 years. Beyond 5 years, between 5% and 10% or more of papers remain uncited. Account must be taken of these time factors in comparing current research with historical patterns. For these reasons, it is sometimes more appropriate to use a fixed 5-year window of papers and citations to compare two periods than to look at the longer-term profile of citations and of uncitedness for a recent year and an historical year. Figure 15 Citation accumulation for a set of 33 journals in the category Materials Science, Biomaterials Discipline factors Citation rates vary between disciplines and fields. For the UK science base as a whole, 10 years produces a general plateau beyond which few additional citations would be expected. On the whole, citations accumulate more rapidly and plateau at a higher level in biological sciences than physical sciences, and natural sciences generally cite at a higher rate than social sciences. Papers are assigned to disciplines (journal categories or research fields) by Thomson Reuters, bringing cognate research areas together. The journal category classification scheme has recently been revised and updated. Before 2007, journals were assigned to the older, well- established Current Contents categories, which were informed by extensive work by Thomson Reuters and with the research community since the early 1960s (4). This scheme has been Page | 276 superseded by the 254 Web of Science journal categories (4), which allow for greater disaggregation for the growing volume of research that is published and abstracted. Papers are allocated according to the journal in which the paper is published. Some journals may be considered to be part of the publication record for more than one research field. As the example below illustrates, the journal Acta Biomaterialia is assigned to two journal categories: Materials Science, Biomaterials and Engineering, Biomedical. Very few papers are not assigned to any research field and as such will not be included in specific analyses using NCI data. The journals included in the Thomson Reuters’ databases and how they are selected are detailed at http://scientific.thomsonreuters.com/mjl/. Some journals with a very diverse content, including the prestigious journals Nature and Science were classified as “Multidisciplinary” in databases created prior to 2007. The papers from these multidisciplinary journals are now reassigned to more specific research fields using an algorithm based on the research area(s) of the references cited by the article. Normalized citation impact Because citations accumulate over time at a rate that is dependent upon the field of research, all analyses must take both field and year into account. In other words, because the absolute citation count for a specific article is influenced by its field and by the year it was published, comparisons of indexed data can only be made after normalizing with reference to these two variables. In calculations of impact for reviews and articles, only citation counts are used, because the document type influences the citation count. For example, a review will often be cited more frequently than an article in the same field, but editorials and meeting abstracts are rarely cited and citation rates for conference proceedings are extremely variable. The most common normalization factors are the average citations per paper for (i) the year and (ii) either the field or the journal in which the paper was published. This normalization is also referred to as “rebasing” the citation count. Impact is therefore most commonly analysed in terms of “normalized citation impact”, or NCI. Figure 16 illustrates how the NCI is calculated at paper level and journal category level. Figure 16 Example of a calculation of the normalized citation impact HRP External Evaluation 2008−2012 Chapter 8 Page | 277 This article in the journal Acta Biomaterialia is assigned to two journal categories: Materials Science, Biomaterials and Engineering, Biomedical. The world average baselines for, as an example, Materials Science, Biomaterials are calculated by summing the citations to all the articles and reviews published worldwide in the journal Acta Biomaterialia and the other 32 journals assigned to this category for each year, and dividing this by the total number of articles and reviews published in the journal category. This gives the category-specific NCI (in the above example the category-specific NCIF for Materials Science, Biomaterials is 5.1 and the category-specific NCIF for Engineering, Biomedical is higher at 6.7). Most papers (nearly two thirds) are assigned to a single journal category, while a minority are assigned to more than five categories. Citation data provided by Thomson Reuters are assigned on an annual census date referred to as the “article time period”. For the majority of publications, the article time period is the same as the year of publication, but for a few publications (especially those published at the end of the calendar year in less mainstream journals), the article time period may vary from the actual year of publication. World average impact data are sourced from the Thomson Reuters National Science Indicators baseline data for 2011. Mean normalized citation impact Research performance has historically been indexed by using average citation impact, usually compared to a world average that accounts for time and discipline. As noted, however, the distribution of citations among papers is highly skewed because many papers are never cited, while a few papers accumulate very large citation counts. That means that an average may be misleading if assumptions are made about the distribution of the underlying data. In fact, almost all research activity metrics are skewed: for research income, PhD numbers and publications, there are many low-activity values and a few exceptionally high values. In reality, therefore, the skewed distribution means that average impact tends to be greater than, and often significantly different from, either the median or mode in the distribution. This should be borne in mind when reviewing analytical outcomes. The average (normalized) citation impact can be calculated at an individual paper level, where it can be associated with more than one journal category. It can also be calculated for a set of papers at any level, from a single country to an individual researcher’s output. In the example above, the average citation impact of the Acta Biomaterialia paper can be expressed as ((5.1 + 6.7)/2) = 5.9. What are uncited papers? It may be a surprise that some journal papers are never subsequently cited after publication, even by their authors. This accounts for about half the total global output for a typical, recent 10-year period. It is not known why papers are not cited. It is likely that a significant proportion of papers remain uncited because they are reporting negative results that are an essential matter of record in their field but make the content less likely to be referenced in other papers. Inevitably, other papers are uncited because their content is trivial or marginal to the mainstream. However, it should not be assumed that this is the case for all such papers. There is variation in non-citation between countries and between fields. For example, relatively more engineering papers tend to remain uncited than papers in other sciences, which is indicative of a disciplinary factor but not a quality factor. While there is also an obvious increase in the likelihood of citation over time, most papers that are going to be cited will be cited within a few years of publication. Page | 278 What is the threshold for “highly cited”? Thomson Reuters has traditionally used the term “highly cited paper” to refer to the world’s 1% of most frequently cited papers, taking into account the year of publication and field. In rough terms, UK papers cited more than eight times as often as the relevant world average would fall into the Thomson Reuters Highly Cited category. About 1–2% of papers (all papers, cited or uncited) typically pass this hurdle. Such a threshold certainly delimits exceptional papers for international comparisons but, in practice, is an onerous marker for more general management purposes. After reviewing the outcomes of a number of analyses, a more relaxed definition has been chosen for descriptive and analytical work. For the purpose of this report, “highly cited papers” have been defined as those papers that belong to the world’s top 10% (top decile) of most cited papers relative to the journal category and year. HRP External Evaluation 2008−2012 Chapter 8 Page | 279 Annex 2: HRP research portfolio HRP supplied Thomson Reuters (Evidence) with a database of HRP publications in Reference Manager. This comprised 3459 records with a unique ID (“Ref ID”), although there were duplicate records. As there may be reasons for duplicated records within the HRP research portfolio, duplicates have not been processed. Matching Matching HRP records to Thomson Reuters unique tags using HRP-provided unique identifiers The HRP research portfolio contained 170 records with Thomson Reuters unique publication identifiers (unique tags) assigned by the Web of Science (UTs). These were validated. It also contained 1670 records with PMIDs – unique publication identifiers assigned by PubMed; 1444 of these records were matched to Thomson Reuters UTs (see Figure 17). Figure 17 Matching HRP records to Thomson Reuters UTs using HRP unique identifiers Matching HRP records to Thomson Reuters unique tags using customized algorithms HRP records were transformed to improve match rates, focusing on titles and journal names in particular. For example, titles were truncated and/or transformed, and special characters were removed. Unmatched journals were grouped by frequency and mapped to standardized Thomson Reuters’ journal names (see Figure 18). Page | 280 Figure 18 Matching HRP records to Thomson Reuters UTs using customized algorithms Matching HRP records to Thomson Reuters unique tags using the Web of Knowledge The final stage of the matching process involved manually searching records on the Web of Knowledge. One hundred and twenty-six records were matched; 25 were deemed ambiguous but satisfactory matches (see Figure 19).4 Figure 19 Matching HRP records to Thomson Reuters UTs using the Web of Knowledge Unmatched records There are two interrelated factors that explain why 1556 records have not matched to the citation databases: (i) document type; and (ii) selectivity of the Web of Knowledge. Document type Figures 20 and 21 show the numbers and proportions of HRP records matched that were not found or abstracted by Reference Manger document types. 4 Twenty-two of these records were validated by HRP. As the three remaining records (relating to two Thomson Reuters UTs) were non-standard journal articles not used in citation analyses, the report is unadjusted, as it would not materially change the results of the analysis. HRP External Evaluation 2008−2012 Chapter 8 Page | 281 Figure 20 HRP records matched, not found and not abstracted by Reference Manager document types (record numbers in brackets), as a percentage of document type Data and analysis: Thomson Reuters (Evidence) Figure 21 HRP records matched, not found and not abstracted by selected Reference Manager document types (record numbers in brackets) Data and analysis: Thomson Reuters (Evidence) Page | 282 Selectivity of the Web of Knowledge Not abstracted Unlike other databases, the Web of Knowledge and underlying databases are selective; that is, the journals abstracted are selected using rigorous editorial and quality criteria. These abstracted journals encompass the majority of significant, frequently cited scientific reports, and an even greater proportion of the scientific research output that is cited. However, it does not abstract all journals. The HRP research portfolio contains a substantial proportion of research output from niche regional journals in obstetrics and gynaecology that are not abstracted. Some examples include the Journal of Reproduction and Contraception, the Chinese Journal of Family Planning and the Chinese Journal of Andrology. This principle also applies to some conferences (for example, the Symposium on Reproductive Health and Infection) and books, which are not yet abstracted by the relatively new Book Citation Index. However, regional journal coverage is expanding, as is the Book Citation Index. There are also document types in the HRP research portfolio, such as WHO technical/special reports, CD-ROMS, news items, theses etc., which may be classified as “non-journal literature” and therefore not likely to be abstracted by the Web of Knowledge. Overall, 1181 records from the HRP research portfolio are not abstracted by the Web of Knowledge. Not found This classification is used where the source of the document is abstracted by the Web of Knowledge but where the particular publication cannot be found. This may be because particular years, volumes, issues or pages have not been abstracted; yet the source overall is abstracted. For example, the Cochrane Database of Systematic Reviews has been abstracted since 2005; therefore, publications before this date are not in the Web of Knowledge. These publications are therefore classified as not found. Overall, 375 records from the HRP research portfolio have not been found in the Web of Knowledge. Author searching A key deliverable of this project was to identify the publications that were not included in the HRP research portfolio. Analysts at Thomson Reuters (Evidence) searched the Web of Knowledge using 72 researcher names supplied by HRP. Address filters were used to help to identify the institutional affiliations of these authors. These were: • HRP – Special Programme of Research, Development and Research Training in Human Reproduction • RHR – Department of Reproductive Health and Research • UNDP – United Nations Development Programme • UNFPA – United Nations Population Fund • WHO – World Health Organization • World Bank • WHO/RHR HRP External Evaluation 2008−2012 Chapter 8 Page | 283 • UNDP-UNFPA-WHO • WHO/HRP • UN/HRP Publications were collated from the start year to the current date, restricted using the HRP- defined address filters. Six hundred and sixty-two unique publications were found (some publications were found more than once as they have more than one HRP researcher). These publications were then compared to the HRP research portfolio to determine those new to the database, using bibliographic information such as title, source, publication year, volume, issue and pages. Two hundred and twenty-seven were identified as new to the database. These publications were validated by the HRP programme manager and returned to Thomson Reuters (Evidence) (see Figure 22). Figure 22 Publications identified by author searching added to the HRP research portfolio Data extraction A total of 2090 records with a Thomson Reuters UT were found, equating to 2004 unique Thomson Reuters UTs. These were submitted for data extraction for the time period 1990– 2011, with citation counts to the end of 2011. As Thomson Reuters UTs enable unique counts to be provided, data are presented here by a unique count (see Figure 23). Page | 284 Figure 23 Publications extracted and not extracted from the Thomson Reuters Web of Science Contents Chapter 1 Chapter 1 Introduction Contents Abbreviations 1 Background to the Special Programme of Research, Development and Research Training in Human Reproduction 2 Previous external evaluations of HRP 3 The current external evaluation of HRP Acknowledgement References Annex 1: Terms of reference for the 2008–2012 external evaluation of HRP Background HRP external evaluation 1990–2002 (presented in 2003) HRP external evaluation 2003–2007 (presented in 2008) Proposal for HRP external evaluation 2008–2012 (to be presented in June 2013) Method of work Annex 2: the Memorandum of Understanding of HRP Chapter 2 Chapter 2Overall assessment of HRP’s relevance and effectiveness Contents Abbreviations 1 Introduction 1.1 A note on language 2 Methods Recommendation 3 Recent challenges 4 The questionnaire 5 Global public health goods 5.1 Priority setting Conclusion Recommendations 5.2 Geographical focus Conclusion Recommendation 5.3 Coordination of research Conclusion Recommendations 5.4 Selected highlights 5.4.1 Promoting family planning Conclusion Recommendation 5.4.2 Maternal and perinatal health Management of the third stage of labour Optimizing maternal and newborn care Maternal “near-miss” 5.4.3 Preventing unsafe abortion Technical and policy guidance on safe abortion Mid-level providers for safe abortion Conclusion 5.4.4 Prevention and control of sexually transmitted and reproductive tract infections including gynecological cancers and infertility Sexually transmitted and other reproductive tract infections Rapid test for syphilis Preventing cervical cancer 5.4.5 Sexual health including adolescence, gender and sexual and reproductive rights A standardized methodology for measuring violence against women Interagency statement on preventing sex selection 5.4.6 The UNFPA country office enquiry Conclusion 5.4.7 Overall conclusions and recommendations Recommendation 6 Comparative advantage 6.1 Respondents’ views on HRP’s comparative advantage Conclusion 7 Norms and standards Conclusion 8 Monitoring of global trends in sexual and reproductive health 8.1 Global maternal mortality estimates. 8.2 Global estimates of unsafe abortion Conclusion 9 The sexual and reproductive health/HIV research agenda 9.1 Selected highlights 9.1.1 Rapid assessment tool for sexual and reproductive health and HIV linkages: evidence review and recommendations The Kesho Bora Study Collaboration with the Global Fund Collaboration with the WHO HIV/AIDS department Collaboration with UNAIDS Conclusion 10 Efficiency and effectiveness of HRP’s communication Recommendation 10.1 Bibliometric analysis of HRP’s papers 10.2 HRP’s channels of communication 10.3 Changes in RHR over the last 5 years and how these have affected HRP’s research communication and uptake Conclusion Recommendations References Chapter 3 Chapter 3 Efficiency and effectiveness of HRP’s governance, management and administration Contents Abbreviations 1 Introduction 2 Funding and fundraising 2.1 Overall funding 2.2 Designated funding Conclusion Recommendation 2.3 Leveraged funding 2.4 Income from royalties 2.5 HRP fundraising initiatives and achievements, 2008 to 2012 Conclusion Recommendation 3 Financial management Conclusion 4 Cosponsorship Conclusion Recommendation 5 Governance 5.1 The Policy and Coordination Committee Recommendations 5.2 The Scientific and Technical Advisory Group Conclusion Recommendation 5.3 The Gender and Rights Advisory Panel Conclusion Recommendation 5.4 The Research Project Review Panel Conclusion Recommendations 5.5 Costs of HRP governance Conclusion Recommendation 6 Management and administration 6.1 Managing for results Did HRP achieve its objectives as laid down in its biennial workplans? Conclusion Recommendation 6.2 Managing research grants How effective is HRP in awarding, processing and monitoring research grants? Conclusion 6.3 Managing research Was the Programme implemented in the most efficient way? Conclusion Recommendation Reference Annex 1: Trends in HRP income from royalties, 2000–2011 Annex 2: Summary of HRP resource-mobilization activities 2008–2012 Annex 3: HRP governance structure Chapter 4 Chapter 4 Evidence generation and synthesis to improve family planning, prevent unsafe abortion and prevent and control sexually transmitted and reproductive tract infections Contents Abbreviations Abstract Introduction Methods Results Family planning: developing a male contraceptive Family planning: getting a new product to market – emergency contraception Family planning: monitoring product safety – hormonal contraception and risk of HIV acquisition Safe abortion services: application of the WHO strategic approach to unsafe abortion – experience in sub-Saharan Africa, eastern Europe and Asia Stage 1: national strategic assessments Stages 2 and 3: country-specific follow-up plans and activities Lessons learnt Controlling sexually transmitted and reproductive tract infections: initiative to eliminate mother-to-child transmission of congenital syphilis Advocacy and technology to support the campaign Progress Effectiveness, efficiency and sustainability Summary Conclusions and recommendations Impact Recommendations Input Process Output/outcomes Summary 1 Introduction 2 Methods 2.1 Phase I: fact finding with WHO programme teams 2.2 Phase II: desk review and consultation with stakeholders 3 Rationale 4 Findings 4.1 General 4.2 Improving family planning: generating new evidence, marketing new products and protecting health and safety 4.2.1 Introduction 4.2.2 Creating new knowledge: developing a male contraceptive Background How male hormonal contraception works Study objectives and methods Trial monitoring, expected side-effects, interim findings and the decision to stop the trial How many men are currently in the study and was the treatment effective? Conclusions: the future of male hormonal contraception development 4.2.3 Getting a new product to market: emergency contraception Background International collaborative efforts to promote and expand access to emergency contraception Emergency contraception: access and impact 4.2.4 Monitoring product safety: hormonal contraception and risk of HIV acquisition Background Methods Summary of the evidence Biological studies Epidemiological studies Acquisition in women who are HIV negative Disease progression in women who are HIV positive Transmission from women who are HIV positive to men who are HIV negative Recommendation 4.2.5 Safe abortion services: application of the WHO strategic approach to unsafe abortion – experience in sub-Saharan Africa, eastern Europe and Asia Background Country experiences Stage 1: national strategic assessments Major findings from assessments in the five African countries and the Republic of Moldova Legal restrictions on abortion/access to contraceptives Systems of clandestine abortion provision had high associated costs for women Inadequate post-abortion care and weaknesses in family planning and sexuality education Religious, social and cultural contexts Stages 2 and 3: country-specific follow-up plans and activities Lessons learnt The situational analysis Flexibility and adaptation Impact Conclusions 4.2.6 Controlling sexually transmitted and reproductive tract infections: initiative to eliminate mother-to-child transmission of congenital syphilis Background Methods Advocacy and technology to support the campaign Case identification and treatment protocols Monitoring outcome/surveillance Progress Effectiveness and efficiency The way forward: sustainability Conclusions 5 Conclusions and recommendations 5.1 Summary 5.2 Impact 5.3 Recommendations References Annex 1: List of persons interviewed and methods of contact Annex 2: Revised guidance: combined hormonal contraceptive use during the postpartum period Recommendation Annex 3: GRADE evidence profiles – systematic reviews of hormonal contraception use and HIV risk A. GRADE evidence profile: hormonal contraception for women at high risk of HIV Use of hormonal contraception in women who are HIV negative B. GRADE evidence profile: risk of HIV transmission among women living with HIV and using hormonal contraceptives Use of hormonal contraception in women who are HIV positive C. GRADE evidence profile: HIV disease progression among women living with HIV and using hormonal contraception Use of hormonal contraception in women who are HIV positive Chapter 5 Chapter 5 Research-capacity strengthening and network building Contents Abbreviations Abstract Introduction Methods Findings Overview Relevance and effectiveness in fulfilment of HRP’s objectives Development of regional networks Outputs of HRP-strengthened research centres are being used to improve health and save lives Sustainability of HRP’s work HRP work through research-capacity strengthening and network building has impacted on women’s reproductive health and children’s health globally The extent to which HRP contributes to global goods The extent to which capacity-strengthening and network-building outputs have reflected the mandate of HRP The comparative advantage of HRP for research-capacity strengthening and network building and as a global leader for research in human reproduction The extent to which HRP work has been relevant to programme countries, particularly to low-income countries The extent to which HRP’s work in research capacity-strengthening is contributing to the achievement of MDGs, ICPD agenda, poverty reduction and women’s health in general Conclusions 1 Introduction 1.1 HRP’s strategy for capacity building for research 1.2 Regional advisory panels 1.3 Objectives of the evaluation for research capacity-strengthening and network building 1.4 Scope of the evaluation 2 Methods 3 Findings 3.1 Overview 3.2 Relevance and effectiveness in fulfilment of HRP’s objectives 3.3 Development of regional networks 3.4 Outputs of HRP-strengthened research centres are being used to improve health and save lives 3.5 Sustainability of HRP’s work 3.6 HRP work through research capacity strengthening and network building has impacted on women’s reproductive health and children’s health globally 3.7 The extent to which HRP contribute to global goods 3.8 The extent to which capacity-strengthening and network-building outputs have reflected the mandate of HRP 3.9 The comparative advantage of HRP for research-capacity strengthening and network building and as global leader for research in human reproduction 3.10 The extent to which HRP work has been relevant to programme countries, particularly to low income countries 3.11 The extent to which HRP’s work in research-capacity strengthening is contributing to the achievement of MDGs, ICPD agenda, poverty reduction and women’s health in general 3.12 Programme strengths and weaknesses 4 Conclusions 5 Lessons learnt 6 Recommendations References Annex 1: Centre recipients of long-term institutional development grants from HRP that were sent a questionnaire by the external evaluator Chapter 6 Chapter 6Strengthening implementation research Contents Abbreviations Abstract Introduction Methods Key findings, comments and conclusions Prioritization of implementation research within the department and the World Health Organization Contribution to implementation research funding The Implementation Research Platform HRP/RHR resources for implementation research High-quality implementation research studies Collaboration with other global implementation research studies Lack of large-scale implementation research studies on priority issues Recommendations 1 Introduction 1.1 What is implementation research? 2 Rationale 2.1 The problem: underuse of life-saving interventions in low- and middle-income countries 2.2 Scaling up life-saving interventions 2.3 Why has this topic has been included in this evaluation? 3 Methods 4 Key findings 4.1 Research governance milestones at HRP related to implementation research 4.1.1 Interteam working group on implementation research. 4.2 Implementation research within the HRP strategic plan for 2010–2015 5 Special initiatives and projects 5.1 The Implementation Research Platform 5.2 The GREAT project (guideline development, research priorities, evidence synthesis, applicability of evidence, transfer of knowledge) 5.2 Global survey on maternal and perinatal health 5.3 The WHO Strategic Approach to strengthening sexual and reproductive health policies and programmes 6 Research projects 6.1 A matched pair cluster-randomized implementation study to measure the effectiveness of an intervention package aiming to decrease perinatal mortality and increase institution-based obstetric care among indigenous populations in Guatemala 6.2 A demonstration project for the implementation of the WHO antenatal care model in Mozambique 6.3 Assessing the acceptability, feasibility and effectiveness of a strategy for improving the quality and safety of maternal/neonatal health care in the health system contexts of four Middle East countries 6.4 Innovations for increasing access to integrated safe delivery, PMTCT and newborn care in rural Uganda 6.5 A cluster randomized controlled trial to evaluate the effectiveness of the clinically integrated RHL evidence-based medicine course 6.6 Comparison of the safety, efficacy, and feasibility of medical abortion by physicians or non-physicians in Nepal 6.7 The effectiveness of antenatal birth plans in increasing skilled care at delivery and after delivery in rural Tanzania 6.8 Comparison of “one-stop” versus “conventional” service on antenatal syphilis screening in Ulaanbaatar, Mongolia 6.9 Collaboration with other agencies or research groups on implementation research initiatives 7 HRP peer-reviewed publications 2008–2012 8 Conclusions 8.1 Positive findings 8.2 Pending issues 8.2.1 Preventing unsafe abortion 8.2.2 Family planning 8.2.3 Maternal/reproductive care 9 Recommendations References Annex 1: Implementation research projects undertaken by HRP/RHR between 2008 and 2012 Annex 2: Publications in peer-reviewed journals related to implementation research projects Publications in peer-reviewed journals relating directly to HRP implementation research projects Chapter 7 Chapter 7 The status of, and opportunities for strengthening, engagement with the private sector and civil society Contents Abbreviations Abstract Introduction and scope of work Key recommendations Regular monitoring of engagement with the private sector and civil society Structures for engaging with the private sector and civil society The way forward Engagement with the private sector and civil society Regular monitoring of engagement with the private sector and civil society Structures for engaging with the private sector and civil society 1 Background 2 Scope of work 3 Approach and methodology 3.1 Approach 3.2 Methodology 4 HRP’s governance structure 5 HRP’s key drivers for engaging with the private sector and civil society 6 HRP’s existing collaborations with the private sector and civil society 6.1 Identification of new products or technologies 6.2 Dissemination of information, tools and policies 6.3 Management of research projects 6.4 Fundraising through commercial activities 7 Findings from analysis of existing collaborations on the basis of the key drivers identified 7.1 Collaborations focused on the identification of new products or technologies 7.2 Collaborations aiming to disseminate information, tools and policies 7.3 Collaborations based on the management of research projects 7.4 Collaborations aiming to carry out fundraising through commercial activities 8 Conclusions and recommendations 8.1 Engagement with the private sector and civil society 8.1.1 Conclusions 8.1.2 Recommendations 8.2 Regular monitoring of engagement with the private sector and civil society 8.2.1 Conclusions 8.2.2 Recommendations 8.3 Structures for engaging with the private sector and civil society 8.3.1 Conclusions 8.3.2 Recommendations 9 The way forward 9.1 Engagement with the private sector and civil society 9.2 Regular monitoring of engagement with the private sector and civil society 9.3 Structures for engaging with the private sector and civil society References Appendix 1: PwC approach and methodology Approach Phase 1: kick off – agree on the terms of reference and scope of work Phase 2: perform – data gathering Phase 3: deliver – data analysis and production of the report Methodology Enhance understanding of HRP from a governance perspective Identify key HRP drivers for engaging in collaborations with the private sector and civil society Review the existing collaborations between HRP and the private sector and civil society Analyse existing collaborations on the basis of the key drivers identified Draw conclusions and propose recommendations for the way forward Chapter 8 Chapter 8Bibliometric analysis Contents Abbreviations Abstract 1 Introduction 1.1 Special Programme of Research, Development and Research Training in Human Reproduction 1.2 Objectives of the report 1.3 About Thomson Reuters 2 Methodology 2.1 Bibliometric data and citation analysis 2.2 Data source 2.3 Definitions used in this report 2.4 Interpretation of data and analyses 3 Baseline bibliometric analysis of HRP research publications 3.1 Key findings 3.2 HRP publication dataset for bibliometric analyses 3.3 In which journals do HRP researchers publish most frequently? 3.4 Is research published by HRP well cited? 3.4.1 In which fields is research published by HRP well cited? 3.4.2 Highly cited research published by HRP 3.5 Trends in the numbers of papers published by HRP researchers 3.6 Trends in the citation impact of papers published by HRP researchers 3.6.1 Trends in highly cited research published by HRP researchers 3.6.2 Summary of changes in the numbers of papers and citation impact between 1990–2007 and 2088–2011 4 International authorship and collaboration 4.1 Key findings 4.2 The principal countries publishing HRP research papers 4.3 Collaboration between countries publishing HRP research papers 4.4 HRP research papers published by researchers based in low-and middle-income countries References Annex 1: Bibliometrics and citation analysis Data source Database categories Assigning papers to addresses Citation counts Time factors Discipline factors Normalized citation impact Mean normalized citation impact What are uncited papers? What is the threshold for “highly cited”? Annex 2: HRP research portfolio Matching Matching HRP records to Thomson Reuters unique tags using HRP-provided unique identifiers Matching HRP records to Thomson Reuters unique tags using customized algorithms Matching HRP records to Thomson Reuters unique tags using the Web of Knowledge Unmatched records Document type Selectivity of the Web of Knowledge Not abstracted Not found Author searching Data extraction Blank Page

View the publication

You are currently offline. Some pages or content may fail to load.