Global FP VAN - Proof of Concept (phase 1) Review

Publication date: 2019

- i - TABLE OF CONTENTS List of Abbreviations . ii List of Figures . iii List of Tables . iv Executive Summary .1 Introduction .3 Background .3 Context – The Starting Point .4 Initial Problem Statement 2017 .4 Initial Vision 2017 .6 Phase 1: Proof of Concept (2017–2018) .7 Vision and Scope .7 Vision .7 Scope .7 Return on Investment .8 Overview – Phase 1 Implementation Progress .8 Two-way Return on Investment (Impact Assessment) . 10 Conclusion . 20 Appendixes . 21 Appendix A: Scorecard Methodology . 21 Background . 21 Objective of the Survey . 21 Survey Methodology . 21 Survey Limitations . 22 Scorecard . 22 - ii - LIST OF ABBREVIATIONS ARTMIS Automated Requisition Tracking Management Information System BMGF Bill & Melinda Gates Foundation CARhs Coordinated Assistance for Reproductive Health Supplies group CCM Country Commodity Manager CDP Country Data Provider CMS Central Medical Store CSP Coordinated Supply Planning group CTP Control Tower Planner DFID UK Department for International Development FP family planning GDSN Global Data Synchronization Network GMM General Membership Meeting GFPVAN/VAN Global Family Planning Visibility and Analytics Network KPI key performance indicator LMIS Logistics Management Information Systems MOS Months of Stock OMS Order Management System PPMR Procurement Planning and Monitoring Report PPT Procurement Planning Tool RFP Request for Proposal RH reproductive health RHI/RHInterchange Reproductive Health Interchange RHSC/Coalition Reproductive Health Supplies Coalition SC Steering Committee TOU Terms of Use UNFPA United Nations Population Fund USAID US Agency for International Development - iii - LIST OF FIGURES Figure 1. Past landscape of automated and manual data flows. .5 Figure 2. VAN conceptual framework. .6 Figure 3. Phase 1 use cases. .8 Figure 4. Road maps of VAN initiative milestones through March 2019. .9 Figure 5. Phase 1 interventions against the original Proof of Concept scope. . 10 Figure 6. VAN central governance structure. . 11 Figure 7. Key performance indicators to measure efficiency, effectiveness, scale, and cost improvements. . 14 Figure 8. Time 1 scorecard progress. . 16 Figure 9. Scorecard and survey alignment and calculations. . 23 - iv - LIST OF TABLES Table 1. Average ratings on the difficulty versus ease of access to supply chain reports for respondents. . 17 Table 2. Average ratings on status transparency and visibility for respondents. . 18 - 1 - EXECUTIVE SUMMARY The Reproductive Health (RH) community has long recognized that limited access to a choice of safe, affordable contraceptives severely undermines efforts to increase contraceptive prevalence, both globally and at the country level. For the Reproductive Health Supplies Coalition (RHSC, or Coalition), a prerequisite to increased supply access must be visibility into global supply chains so teams can improve supply chain performance. Put simply, “You cannot manage what you cannot see.” Governments and global procurers need timely visibility for effective decision-making, for estimating supply needs, for acting when product arrives, and for advocating if funding falls short. This means making access to planned orders, to shipment progress from manufacturer to country, and to country- level inventory and demand data available to a varied audience of supply chain players. In 2016, members of the RH community asked the RHSC to take the lead in defining and operationalizing a more coherent and efficient way to gather and use data for upstream supply chain decision- making. The vision was to act on that request and put in place a Global Family Planning Visibility and Analytics Network (GFPVAN, or VAN). The VAN is meant to bring together people, processes, policy and technology to transform the way our community makes supply chain decisions. It offers a platform to collectively estimate and prioritize supply needs, and people and processes to act when supply imbalances loom, and policy to govern data sharing and use. Eventually, a well-functioning VAN will lead to more timely and cost- effective delivery of commodities; more women reached with the right product at the right time; and a better allocation of limited health resources. The VAN is not just another data tool. It is much more powerful than that. It is a “network.” Networks bring people and information together in new ways to transform how we do things and make life easier. Think of the ATM network that makes it easier to get money. Or the Facebook network that makes it easier to keep in touch with friends. Similarly, Visibility and Analytics Networks are supposed to make it easier to get products—in this case, family planning (FP) products. Given their complexity, the sustainability of these networks depends on their relevancy to users and, in return, user willingness to pay to sustain them. They need to offer connections and opportunities that - 2 - would not exist otherwise and that, once tried, users can no longer imagine going without. At the heart of such a network is a structural bond of community users that allows them to transform how they give-and-take from each other and interact. That structural bond must be managed by a central governance structure, supported by flexible technology that can build on itself, and built on a membership and contracting model that can grow. With the end of the VAN Proof of Concept (phase 1), it is important to look backwards and measure the success and return on investment of phase 1 in order to build a disciplined vision for network growth. This document provides a summary of the learnings to date. It reminds the reader what we aimed to achieve in the first year by including a retrospective view of the initial problem statement followed by a description of progress against the phase 1 vision and scope. Return on investment is then analyzed in two areas: (1) foundational impacts across people, processes, policy, and technology that are a community good, no matter the system used or the long-term status of the VAN; and (2) scorecard progress against efficiency, effectiveness, scale, and cost. The conclusion is that phase 1 successfully proved the concept of the VAN and has laid the foundation and groundwork for continued investment and growth in future phases. - 3 - INTRODUCTION Background The Reproductive Health Supplies Coalition (RHSC, or Coalition) is a global partnership of more than 475 public, private, and nongovernmental organizations dedicated to ensuring that all people in low- and middle-income countries can choose, obtain, and use affordable, high-quality supplies to ensure their better reproductive health (RH). The Coalition brings together diverse agencies and groups with critical roles in providing contraceptives and other RH supplies. These include multilateral and bilateral organizations, private foundations, governments, civil society, and private-sector representatives. The RH community has long recognized that limited access to a choice of safe, affordable contraceptives severely undermines efforts to increase contraceptive prevalence, both globally and at the country level. For the Coalition, a prerequisite to increased supply access must be visibility into global supply chains so teams can improve supply chain performance. Put simply, “You cannot manage what you cannot see.” Governments and global procurers need timely visibility for effective decision-making, for estimating supply needs, for acting when product arrives, and for advocating if funding falls short. This means making access to planned orders, to shipment progress from manufacturer to country, and to country-level inventory and demand data available to a varied audience. The Coalition and its members have over ten years of experience putting in place the people, processes, policies, and tools to promote supply chain visibility and collaborative decision-making between governments and global procurers. To advance this work, the RH community asked RHSC to lead and manage the establishment of a Global Family Planning Visibility and Analytics Network (GFPVAN, or VAN), or a virtual, collaborative platform where community members can simultaneously access a shared set of data to collectively make supply chain decisions. From the people side, it is meant to link procurers, manufacturers, shippers, and countries in an active network focused on product flow into countries. In terms of technology, the platform needs to capture data from many sources, facilitate data harmonization, and consolidate tools for network members to use. New processes should transform how these members interact, analyze data, and make decisions. Harmonized policies are needed to govern data sharing and use. The VAN is meant to offer a platform to collectively estimate and prioritize supply needs so that members can take action when supply imbalances loom and advocate for funding when necessary. Eventually, a well-functioning VAN will lead to more timely and cost-effective delivery of commodities; more women reached with the right product at the right time; and a better allocation of limited health resources. At the July 2017 London Family Planning Summit, key stakeholders announced their commitment to establishing a Global Family Planning VAN. RHSC agreed to host and manage its implementation. The Bill & Melinda Gates Foundation (BMGF) and the UK Department for International Development (DFID) committed to providing seed money. And the US Agency for International Development (USAID) and the United Nations Population Fund (UNFPA) agreed to provide the human resources to design, test, and use the platform. These players, plus one user representative, form the Steering Committee (SC), which is mandated to set the vision and oversee the implementation. The first major investment was made by the Gates Foundation at the end of August 2017 as catalytic funding to kick-start work. Phase 1, or the Proof of Concept phase, is defined as starting in-line with that investment. Key milestones include scope and road map approval in October 2017, system selection and contracting by May 2018, live platform launch on January 22, 2019, and the development of key process-improvement documents in February 2019. - 4 - CONTEXT – THE STARTING POINT Initial Problem Statement 2017 A prerequisite to increased supply access must be visibility into global supply chains so teams can improve supply chain performance. Governments and donors need visibility for effective decision-making, for estimating supply needs, for acting when product arrives, and for advocating if funding falls short. Given the centrality of supply chain visibility to the RH’s community work, the RHSC has led the development of virtually every diagnostic tool in the RH-commodities space. These tools have been developed with the express purpose of providing greater supply chain visibility to be able to confront the obstacles that impede the flow of goods and supplies from manufacturer to country central medical stores: • In 2004, the RH community consolidated into a single online platform the volume of contraceptives supplies procured by the donor community, ultimately achieving up to 80 percent coverage. RHSC launched the Reproductive Health Interchange (RHInterchange, or RHI), which remains to this day the most widely used, authoritative source of total commodity volumes and pricing. • In 2005, the RH community grappled with the growing problem of immediate stock crises: overstock, understock, and expiration. RHSC launched the Coordinated Assistance for Reproductive Health (CARhs) group, which, five years later in 2009, would prompt the development and launch of the Procurement Planning and Monitoring Report (PPMR)—which remains the core source of information on stock levels at the country level. • In 2012, the RHSC flipped the need for managing exceptions by embarking on a new initiative to ensure better and more aligned procurement from the outset and active monitoring of country stock status over the medium-term. The Coordinated Supply Planning (CSP) group was launched. • In 2014, the FP Market Report was established with buy-in from more than ten RHSC members to reassess the volume of commodity flows by concentrating on data from the perspective of manufacturers and not institutional procurers. • In 2016, RHSC completed an assessment of the supply requirements (and associated costs) of meeting the growing demand for FP. When assessed in 2017, the efficient management and long-term future of these tools were unsure. The RH community’s ability to tap different data sources, deliver timely data, and make quick decisions was becoming increasingly constrained. There were several reasons for this. First, many of the tools were developed at different points in time and, ultimately, in isolation from one another, with different data sources and underlying software. The diverse and siloed nature of these data platforms made triangulation of information time-consuming and hugely inefficient. In certain cases, data cleaning had to be done manually. - 5 - In 2017, the data that was used for driving decisions across the RH community was hosted in a number of separate systems across USAID, UNFPA, RHSC partners, country ministries of health, and nongovernmental organizations (see Figure 1). Each organization had separate product and process definitions. Additionally, there was wide variability in the technology capacity and sophistication. Few of these applications communicated directly with one another. The effect of this was an intensely manual process to reconcile data from each system to understand the impact of a change or recommendation. The process of manually collecting, aggregating, normalizing, and aligning data slowed the effectiveness of staff and limited the ability to scale analysis to more countries and commodities. Second, some of the IT systems were simply outdated or had outlived their platforms. The RHI, one of the first Coalition-hosted tools, was designed long before pharmaceutical manufacturers became Coalition members. As a result, data flows were channeled through procurers, a necessary but ultimately inefficient and potentially duplicative strategy. Third, many of the tools were managed by different partners, who managed them within a defined scope and within their financial constraints. As the community expanded its coverage to more countries, products, and manufacturers, sometimes the managing partner was unable to keep pace due to scope and financial barriers. A centrally managed community tool was needed to expand the scope and pool resources. And fourth, bilateral data-sharing agreements limited the use of data for community-wide decision-making and made it cumbersome and challenging to share the data in collaborative forums. Multilateral, collaborative agreements comprising role-based data-use definitions were needed to overcome growing proprietorship over data. For more details on the 2017 description of the problem, please see Section 7 of the RFP Addendum that was published in January 2018 for vendors to use in preparing a Request for Proposal (RFP) response. Figure 1. Past landscape of automated and manual data flows. Abbreviations: ARTMIS, Automated Requisition Tracking Management Information System; CCM, Country Commodity Manager; CSP, Coordinated Supply Planning; LMIS, Logistics Management Information Systems; OMS, Order Management System; PPMR, Procurement Planning and Monitoring Report; PPT, Procurement Planning Tool; RH, Reproductive Health; UNFPA, United Nations Population Fund. - 6 - Initial Vision 2017 Based on the initial problem statement outlined above, members of the RH community asked the RHSC to take the lead in defining and operationalizing a more coherent and efficient way to gather and use data for upstream supply chain decision-making—potentially to be used by the health community as a whole. The 2017 vision was for RHSC to act on that request and put in place a VAN that brings together people, processes, policy, and technology to transform the way our community makes supply chain decisions (Figure 2). From the people side, it is meant to link procurers, manufacturers, shippers, and countries in an active network focused on product flow into countries. In terms of technology, the platform needed to capture data from many sources, facilitate data harmonization, and consolidate tools for network members to use. New processes would transform how these members interact, analyze data, and make decisions. Harmonized policies were needed to govern data sharing and use. As envisioned in 2017, the VAN was meant to offer a platform to collectively estimate and prioritize supply needs so that members can take action when supply imbalances loom and advocate for funding when necessary. An important goal of the VAN, therefore, is to better aggregate, share, and align country-demand data with production, procurement, and funding to avoid stock imbalances and maximize use of limited resources to ensure continuous availability and choice of FP products in developing countries. The bigger vision was and still is that eventually a well-functioning VAN will lead to more timely and cost-effective delivery of commodities; more women reached with the right product at the right time; and a better allocation of limited health resources. For more details on the proposed vision in 2017, please see Section 8 of the RFP Addendum that was published in January 2018 for vendors to use in preparing an RFP response. Figure 2. VAN conceptual framework. - 7 - PHASE 1: PROOF OF CONCEPT (2017–2018) Vision and Scope Vision Phase 1 was designed to be a Proof of Concept. It was about demonstrating on a small scale that the RH community could put in place a VAN with established governance (people), processes, policies, and technology that could ultimately lead to more efficient and effective collaborative supply-and-demand planning, order management and tracking, and exception-based management than the current or previous systems. It was also designed to be a learning phase. So, built into it was a high tolerance for flexibility, for reflection, and for occasional dead ends. The Steering Committee accepted that it was not sure how things would work out, or where roadblocks would appear, but that that did not mean failure if the learnings were documented. One of the key aims of phase 1 was to see what would happen if the community moved forward aggressively and “held its feet to the fire,” put in place key foundations to guide future decision-making, identified what worked versus the road blocks, and documented progress so that the FP community would be better able to pursue its longer-term objectives. Scope The scope of phase 1 was formally endorsed during the in-person Steering Committee meeting that took place on October 11 and 12, 2017, at UNFPA headquarters in New York City. There, the Steering Committee voted and agreed on the following scope: • Two pilot countries (Nigeria and Malawi would later confirm their interest). • Two products (implants and oral contraceptives). • Four manufacturers (Bayer AG, Merck Sharp & Dohme Corp., Mylan Laboratories Ltd., and Shanghai Dahua Pharmaceuticals Co, Ltd. would later confirm their interest). • Six initial intervention use cases relating to sustainability, interoperability, inbound visibility (shipments and orders), and planning and adjusting (inventory and supply plans). Figure 3 outlines the exact “what if” statements associated with each use case below. These use cases represent transformation interventions in over 75 percent of the existing supply chain ecosystem; only the initial supply-request-and-approval process is missing. - 8 - Figure 3. Phase 1 use cases. Abbreviations: CARhs, Coordinated Assistance for Reproductive Health Supplies group; CMS, Central Medical Store; CSP, Coordinated Supply Planning group; GFPVAN, Global Family Planning Visibility and Analytics Network; LMIS, Logistics Management Information Systems. The scope parameters were documented and discussed with stakeholders and interested bidders who participated in the collaborative requirement definition process. The scope is further defined in the RFP Addendum that was published in January 2018. Return on Investment For the phase 1 Proof of Concept, the return on investment has to do with impact in two areas: (1) foundational impacts across people, processes, policy, and technology that are a community good, no matter the system used or the long-term status of the VAN; and (2) scorecard progress against efficiency, effectiveness, scale, and cost. The section below will outline an overview of intervention progress against the original Proof of Concept scope and then will go into an analysis of return on investment against the two elements above. Overview – Phase 1 Implementation Progress Phase 1 was designed to have an aggressive timeline to see what could happen if we “held each other’s feet to the fire” with about one milestone a month and used initial quick wins to encourage continued momentum. The VAN initiative met all the milestones outlined in the Figure 4 road maps below. - 9 - Figure 4. Road maps of VAN initiative milestones through March 2019. Abbreviations: GDSN, Global Data Synchronization Network; GMM, General Membership Meeting; RFP, Request for Proposal; RHSC, Reproductive Health Supplies Coalition; SC, Steering Committee. Overall implementation of phase 1 interventions against the original Proof of Concept scope stayed on track or surpassed targets in most places, as outlined in the Figure 5 below. There were also two places where implementation interventions surpassed expectations. Both have to do with exceeding visibility expectations around the breadth of data already loaded and visible on the platform. The scope of data visibility for two product families and two countries was surpassed, with order and shipment data flowing into the system for all product families, except condoms; order and shipment data from January 1, 2017, flowing in for 105 countries; and inventory data from January 1, 2017, flowing in from 37 PPMR countries. Under interoperability, the breadth of the types of data that can be ingested was also exceeded. Any country that is using PipeLine for supply planning can load planning data into the system via a drag-and-drop interface, and any manufacturer can provide shipment updates directly to the system via a drag-and-drop interface. - 10 - Figure 5. Phase 1 interventions against the original Proof of Concept scope. Abbreviations: GDSN, Global Data Synchronization Network; GFPVAN, Global Family Planning Visibility and Analytics Network; PPMR, Procurement Planning and Monitoring Report; RH(I), Reproductive Health (Interchange); TOU, Terms of Use. Two-way Return on Investment (Impact Assessment) What has been the impact of keeping on track with the aggressive timeline and the intervention progress against the scope? This section aims to answer this question from two different perspectives. First, it looks at the long- term impact expected from a series of foundational building blocks put in place. In this section, it outlines best practices and lessons learned that will be shared with the community to avoid the cost of recreating the wheel in the future. Second, it uses a scorecard to measure initial impact against KPIs (key performance indicator). - 11 - Foundational Impacts An important part of phase 1 was proving that the FP community could agree and put in place the foundational elements necessary to establish a VAN. The foundational elements include a central governance structure for improved cross-organization, coordinated action for FP supply chain management (people); a transformed vision for how to efficiently and effectively coordinate supply chain management at the global level (process); a multilateral policy governing use of the platform and its data (policy); and a system selection approach that is robust enough that many different types of entities can buy in to the network in the future. These elements were not only foundational to phase 1 of the initiative but also are the building blocks to the sustainability of the initiative and any future global VAN-type efforts, even if those efforts are separate from the VAN. These foundational elements represent a series of community goods that have positively impacted our community. As such, they represent positive long-term returns on investment or impacts, and elements from all four below will be written up for dissemination. People: A central governance structure Having the right governance structure is key to an initiative like the VAN that requires a collective mindset. The Coalition was asked by the community to host and manage the VAN, in part because of its convening power and experience putting in place governance and management structures. In the case of the VAN, the Coalition deliberately chose a management structure similar to that of RHSC, especially since the VAN, like the Coalition, brings together partners, but without any long-term funding commitments. As you can see from Figure 6, the VAN is governed by a Steering Committee, analogous to the RHSC’s own Executive Committee. The Management Unit functions along the lines of the Coalition Secretariat insofar as it implements day-to-day tasks. It was defined by the Steering Committee to include a Director, Supply Chain Data & Technology Specialist, and consultants, as needed. Finally, the four Task Forces operate in similar fashion to the Coalition’s Working Groups. Their roles were identified early on, but the content of the work evolves according to shifting needs and changes in membership. Figure 6. VAN central governance structure. Returning to the Steering Committee, it stands at the summit of the governance structure. Its membership is made up of the key financial supporters and change agents across the Gates Foundation, DFID, UNFPA, USAID, and RHSC. It also includes John Snow, Inc. as a user representative. The group has met every fortnight since December 2016. - 12 - There is a Terms of Reference that was written by the Steering Committee. The Steering Committee makes all major decisions following the timeline in the formalized road maps. To date, every major milestone decision (including scope, use case, requirements, country/product/manufacturer selection, system selection, indicator establishment, user acceptance testing, etc.) has followed a clear approval process. In addition, four active Task Forces were established in phase 1. The Super User Group consists of FP supply chain actors, including procurers, data analysts, manufacturers, and countries, to inform the process for collaborative supply chain management. The other three groups include the Data Sharing Task Force, focused on data-sharing policy; the Data Management Task Force, focused on harmonizing data structures; and the Technology Task Force, focused on the system and integrations. The Task Forces play an essential role in securing broad-based multisectoral input into decision-making processes; fostering a joint understanding of the VAN vision and approach; and giving those involved some skin in the game. Not only do Task Force members offer their technical expertise and help craft key documents for approval, in some instance they are also asked to go a step further and endorse items, such as key requirements, before they are approved. This level of involvement has increased ownership across Task Force members. Regardless of the future of the VAN initiative, a blueprint has been defined for an effective central governance structure to promote coordinated supply chain management moving forward. The design and project management approach have been established and are being documented in a mini-case study that will be disseminated publicly. This is an important community good for the long-term. Process: A transformed vision for coordinated supply chain management In order to define a clear set of business requirements for vendors to use in proposing a technology solution and configuration design, those involved in operational coordinated supply chain management from the CARhs and CSP groups participated in process mapping and collaborative requirement setting. They had to envision a new, more efficient and effective way of working together. This community-endorsed vision for process improvement is outlined in the current and future state sections, as well as the requirements section, of the RFP Addendum. The documented vision and requirements in the RFP Addendum are a public good for future iteration. Those involved on the operational side also formed a Super User Group, as outlined above, and they developed two other long-term foundational elements. First, the Super User Group was able to forge a standardized set of KPIs to measure the performance of collaborative supply chain operations in the VAN that previously were measured in different, subjective ways across the supply chain. The “workflow” design of the VAN ticketing module allowed the Super User Group to define indicators in the system to track who is doing what in the system and how people work together. The advantage is that the community can now objectively measure which supply chain activities are taking the longest and where the barriers are—a critical step in improving efficiency and effectiveness. These community-approved KPIs, in tandem with the “workflow” design, are a community good and together form an important part of the process-improvement blueprint for the merging of CSP and CARhs. Second, the Super User Group also worked throughout User Acceptance Testing and the initial launch of the platform to define the new job descriptions and standard operating procedures required to merge CSP and CARhs into a more efficient and effective Logistics Management Unit, or “Control Tower.” The defined roles, job descriptions, organizational charts, and standard operating procedures are being documented as a community good and will go through initial and future iterations as needed. - 13 - Policy: A multilateral policy governing platform and data use A more efficient VAN demands streamlined legal parameters for ensuring supply chain visibility. Traditionally, the issue of data confidentiality has been handled through negotiated, bilateral agreements, which can often make it legally confusing and logistically cumbersome for multilateral groups seeking to share data. The VAN resolved such barriers through the development of a transparent, multilateral Terms of Use (TOU) agreement hinging on a robust, secure, role-based data security model. VAN users must agree to the terms before logging into the platform and using the data. The TOU approach is one that can be used by other community members working on multilateral, collaboration initiatives. Instead of having to reinvent the wheel, they can build on the learnings from the VAN. The lessons learned and guidance will be written up for dissemination. Technology: A robust system selection approach that allows buy-in from different entities How did the various VAN stakeholders, with their individual needs and priorities, choose a single technology vendor to satisfy everyone? Stakeholders and bidding vendors used an interactive, collaborative web-based software called Basecamp, which allowed partners to simultaneously see different documents as they were developed, formulate questions, share feedback, and develop final system requirements, thereby forging a collective “ask.” These requirements were then validated by 23 community endorsers and Steering Committee members and used in proposal development and scoring. The rigorous tendering process ultimately led to the selection of E2open as the technology vendor. The transparent and collaborative process was designed so that government and multilateral entities—like USAID, UNFPA, country governments, and others, which require rigorous tendering processes—could eventually buy in to the platform. This system selection approach had an important foundational impact because it provided a blueprint for others trying to do similar complex multilateral RFP processes, and it provided a sustainable means for others to pay into the platform moving forward. Per the Gartner 2019 report on Multienterprise Supply Chain Business Networks, “only a few companies have had success with multienterprise process management because organizations must be willing to work in partnership— sharing information, redefining processes and implementing mutually beneficial governance.”1 The VAN initiative is built on partnership and has established the foundational elements necessary across the people, processes, policy, and technology above for sharing information, redefining processes, and implementing governance. Scorecard Approach and Analysis The expectation is that putting in place a VAN with established governance (people), processes, policies, and technology will ultimately lead to more efficient and effective collaborative supply chain management, allowing scale-up coverage to more countries and products without costing more. In August 2018, the Super User Group and Steering Committee agreed on a set of specific KPIs to measure efficiency, effectiveness, scale, and cost improvements (see Figure 7). Demonstrating the magnitude of these improvements is critical to establishing the value proposition of the VAN. 1 Gartner website, https://www.gartner.com/doc/reprints?id=1-5SEIK3R&ct=181115&st=sb. - 14 - Figure 7. Key performance indicators to measure efficiency, effectiveness, scale, and cost improvements. Abbreviations: CDP, Country Data Provider; CSP, Coordinated Supply Planning group; CTP, Control Tower Planner; FP, family planning, MOS, Months of Stock; PPMR, Procurement Planning and Monitoring Report; RHI, Reproductive Health Interchange; VAN, Global FP Visibility and Analytics Network. Two different measurement methodologies were also approved. First, the KPIs include objective, quantitative indicators that will be measured by the VAN platform. Second, there are also subjective indicators that are measured by a longitudinal evaluation to show change over time. The survey is, and will continue to be, administered in alignment with key milestones throughout the course of initial VAN implementation. Survey questions have been structured to gain insight into key stakeholders’ experience with the VAN, including both the technology system and data, and the piloted collaborative supply chain processes. - 15 - The scorecard has been developed to pull together a snapshot view of key KPIs across both measurement methodologies, and other relevant statistics. Since the VAN is meant to bring together people, processes, policy, and technology to transform the way the FP community makes supply chain decisions, the scorecard is designed to measure progress across those conceptual framework elements, as well as the value propositions related to efficiency, effectiveness, scale, and cost. It offers quick and easy analysis of return on investment by key stakeholders. For more information on the scorecard methodology, see Appendix A. This section uses existing data to analyze scorecard progress (see Figure 8). The data time frame for time 1 is December 2018 for the KPIs and February 2019 for the other statistics. Certain KPIs and statistics are grayed out for time 1, given that the VAN platform was only launched in live production mode on January 22, 2019. As described above, the longitudinal evaluation survey is administered around key milestones and gathers data related to the subjective KPIs. The available data for phase 1 was the survey administered in December 2018. Many of the scorecard survey indicators are grayed out to clarify that the platform was not live, and respondents could not yet reliably assess the results. The Process Improvement and System Usability percentages are shared, however, because respondents were able to assess this based on their User Acceptance Testing. The next survey is scheduled for several months after the launch of the live platform and training to allow users to have enough comparison to report in responses. As for the objective KPIs in the scorecard, these are measured by the platform, and since the platform just went live in January, there is not yet enough data to measure. They will be included in the next scorecard review. A note on the scorecard frame: Given that the FP community’s complete requisition-to-arrival cycle can take up to a year, supply chain impact (lead time, on-time and in full delivery, etc.) is only apparent after a year. For this reason, the scorecard does not yet reflect these macro-level goals. They will be included in the future. Figure 8. Time 1 scorecard progress.A note on the scorecard frame: Given that the FP community’s complete requisition-to-arrival cycle can take up to a year, supply chain impact (lead time, on-time and in full delivery, etc.) is only apparent after a year. For this reason, the scorecard does not yet reflect these macro-level goals. They will be included in the future. - 16 - Figure 8. Time 1 scorecard progress. - 17 - Overall, the inefficiency and ineffectiveness pain felt by those using current processes for collaborative FP supply chain processes has reduced from 5.9 to 4.4 on a 10-point pain scale. Since this is only one data point in time, it will be important to continue to monitor the trend and see if it continues a downward trajectory over time. In terms of people and policy, the scorecard offers numerical support to the return on investment described in the foundational impact above. The scorecard highlights the breadth of community involvement in phase 1. Buy-in has been achieved on all key decisions across six Steering Committee member-organizations, four of which vote, and 16 Task Force member-organizations. As of February 2019, there are 21 active, individual VAN members who have signed in, accepted the TOU, and started using the platform. As for efficiency, effectiveness, cost, and scale associated with the process side, available data have been included in the scorecard and are further explained below. Efficiency Under efficiency, most scorecard KPIs could not be measured yet. The scorecard does include baseline and time 1 responses for the Process Improvement indicator, however. Of those that responded based on their User Acceptance Testing experience, 75 percent indicated that it was "somewhat easy" or "very easy" to access the supply chain reports and data analyses needed for review with the FP community, versus only 19 percent at baseline. In comparing average ratings on the difficulty versus ease of access to supply chain reports and data analysis for the respondents, the average ratings improved for all actors, per Table 1 below. Table 1. Average ratings on the difficulty versus ease of access to supply chain reports for respondents. In the future, the expectation is to see further positive statistics in the efficiency section of the scorecard, due to key efficiency gains established in phase 1: • Process Improvement. Harmonized Product Master between UNFPA and USAID that includes known Global Trade Item Number attributes and United Nations Standard Products and Services Codes. • Time Savings and Visibility. Established single system for tracking orders, shipments, inventory, and supply plans across donors. • Visibility. Expanded breadth of data types able to be loaded, including the following: o Any country that is using PipeLine for supply planning can load planning data into the system via a drag- and-drop interface. o Any manufacturer can provide updates directly to the system via a drag-and-drop interface. • Time Savings, Visibility, and Process Improvement. Established Global Data Synchronization Network (GDSN) subscription, allowing the VAN to pull master data directly from the GDSN data pool for subscribed manufacturers in the future. • Time Savings. Tested integration automation on pulling stock/inventory information directly from country electronic logistics management systems. Actor/Role Baseline (n=6) Pilot 1 (n=4 ) Control Tower Data Analyst 2.0 4.0 Control Tower Administrator 2.0 3.5 Global Procurer 1.3 4.0 - 18 - • Process Improvement. Demonstrated ability to merge CSP and CARhs into one Control Tower Logistics Management Unit, thereby leading to future time savings. These are all critical foundational efficiency gains for the future. Since the platform just launched, and process change design is still underway, users are currently working in the old systems, while trying out and developing the new processes. The result is more work now for a reduction in the future. Effectiveness Under effectiveness, it is important to look at current users’ future expectations of the VAN, based on their involvement to date. Eighty-five percent of respondents expect in the future that the VAN will allow them to make more timely and specific supply chain recommendations and decisions regarding Nigeria and Malawi compared to before. This indicator measures better coordination and will be important to follow moving forward to see if that expectation grows as data access and visibility increase. System usability and workflow improvements are important when looking at return on investment. It is expected that improved usability and work status transparency will ultimately have a hand in improving supply chain actions and access to product. Twelve respondents were able to respond based on their experience in staging and User Acceptance Testing. Forty-two percent of respondents currently “agree” or “strongly agree” that the status and progress of the collaborative FP supply planning process is transparent and visible at all times, compared to 27 percent at baseline. In comparing average ratings on status transparency and visibility for the respondents, with both a baseline and time 1 numerical value (see Table 2), average ratings improved for all actors. Table 2. Average ratings on status transparency and visibility for respondents. The expectation is that the workflow ticketing module, plus the alerts and notifications on performance issues and changes, will make the system more user-friendly than current processes. This indicator is moving in the right direction and should increase as respondents interact on the live platform. The other critical foundational elements that are driving current and expected future effectiveness gains are described above in the Foundational Impacts section (Page 11). Scale Having more efficient and effective collaborative supply chain processes is meant to allow the community to engage in collaborative supply chain management across more countries and products than was previously possible without adding additional costs. In the longitudinal evaluation, 63 percent of those who responded agreed that that is “likely” or “very likely.” That is more than the majority, but there is hesitancy. Actor/Role Baseline (n=15) Pilot 1 (n=12) Control Tower Data Analyst 4.0 4.5 Control Tower Administrator 1.0 4.5 Global Procurer 2.0 3.0 Control Tower Planner 2.1 2.8 - 19 - Twenty-five percent are still unsure on the two scale indicators because they are waiting to see how processes change with the technology, which makes sense given that the platform was not yet live when the survey was administered. This indicator is expected to increase once respondents are on the live platform. In the meantime, what is agreed is that more visibility and access to data are necessary for any movement on scale and reduction in the inefficiencies of the old processes. Under phase 1, the VAN exceeded visibility expectations around the breadth of data already loaded and visible on the platform, with 105 countries’ order and shipment data loaded, and 37 countries’ PPMR inventory loaded. In addition, order and shipment data have been loaded for all USAID and UNFPA FP product families, except condoms. The increased visibility, coupled with the process improvements and system usability, is expected to lead to increased positive statistical responses in the future. Cost The Management Unit has made a concerted effort to gather quantitative understating of baseline soft and hard costs related to RHI, PPMR, CSP, and CARhs for comparison over time to document cost savings. The challenge has been gaining access to the data, given the number of years and changes in personnel since the tools and groups were established. Enough of the CSP and CARhs soft costs have been gathered to compare cost savings as they merge moving forward. Cost comparisons due to process change will start to be measured in the future, once the two groups officially merge. In the meantime, it is useful to look to the survey and the expectations of users. Ninety-seven percent of respondents "agree" or "strongly agree" that current systems (CSP Online, RHI, PPMR) could be retired if the VAN covered the required countries and products. The ability to replace all these separate tools, with their separate maintenance and management costs, is an expected cost savings. The CSP Online Tool was already retired in phase 1. Phase 1 also established the feasibility to replace RHI, both in terms of secured data access via the VAN platform and public access via the Reproductive Health Data Visualization (RH Viz) dashboards, assuming the condom product master issues are finalized with UNFPA. The VAN initiative will continue to explore how to document these cost savings. - 20 - Conclusion With the end of the VAN Proof of Concept (phase 1), it is important to look backwards and measure the success and return on investment of phase 1 in order to build a disciplined vision for network growth. This document provides a summary of the learnings and impact to date. Given the foundational impacts, the scorecard progress and the survey findings, the conclusion is that phase 1 successfully proved the concept of the VAN and has laid the foundation and groundwork for continued investment and growth in future phases. The VAN Steering Committee has approved and committed to support the continued growth of the VAN and learning in phase 2 (May 2019-April 2020). The vision, scope, roadmap and review of phase 2 will be shared moving forward. - 21 - APPENDIXES Appendix A: Scorecard Methodology Background The business case expectation is that putting in place a VAN with established governance (people), processes, policies, and technology will ultimately lead to more efficient and effective collaborative supply chain management, allowing scale-up coverage to more countries and products without costing more. In August 2018, the Super User Group and Steering Committee agreed on a set of specific KPIs to measure efficiency, effectiveness, scale, and cost improvements (see Figure 7). Demonstrating the magnitude of these improvements is critical to establishing the value proposition of the VAN. Two different measurement methodologies were also approved. First, the KPIs include objective, quantitative indicators that will be measured by the VAN platform. Second, there are also subjective indicators that are measured by a longitudinal evaluation to show change over time. The survey is, and will continue to be, administered in alignment with key milestones throughout the course of initial VAN activity implementation. Survey questions have been structured to gain insight into key stakeholders’ experience with the VAN, including both the technology system and data and the piloted collaborative supply chain processes. Objective of the Survey The current longitudinal evaluation seeks to better understand key stakeholders’ experience with coordinated supply chain management before and after the introduction of the VAN. Specifically, it measures key supply chain stakeholders’ perspectives over time regarding the extent to which putting in place a VAN results in more efficient and effective collaborative FP supply chain management, allowing scale-up coverage to more countries and products. Survey Methodology In September 2018, a short 15-item structured questionnaire was developed by the Management Unit staff and sent out electronically, via Survey Monkey, to 37 individuals who play a critical role in collaborative FP supply chain management. Thirty-four submitted a completed survey. In November 2018, the same survey questionnaire was sent again to a subset of 20 individuals trained on the VAN platform. All submitted a completed survey. All respondents were asked to consider their experience and respond to the questions with their assigned role in mind at the time of the survey. If a particular question referenced processes, tools, or data that they were not involved with, respondents were asked to respond N/A. Differences between the baseline and second measurement (time 1) were analyzed using Excel. Specifically, for each question we calculated the change in respondents’ average rating, excluding from the denominator all “N/A” responses. Additionally, we compared pain-scale rating averages for the supply chain management “roles” for which we had both a baseline and time 1 numerical value. - 22 - Survey Limitations First, although the survey was designed to collect information from the same group of individuals at regular intervals, the baseline and time 1 samples were not identical due to a number of factors. Fourteen individuals who completed the baseline survey were not included in the time 1 survey, as they had not yet been trained on using the VAN platform. We expect these actors to participate in future measurements, as the survey continues to be administered in alignment with key VAN milestones. Also, one respondent changed organization and role vis à vis the VAN between the baseline and time 1 measurements. Nevertheless, we included the respondent’s answers in the survey analysis. Four individuals who no longer play a key role in the VAN were excluded from the analysis and will continue to be excluded moving forward. Second, given that the VAN platform had not been launched in live production at the time of the December 2018 survey, a number of respondents could not yet fully play their assigned role in the platform and selected “N/A” for many of their responses. We excluded the “N/A” from the denominator when comparing average scores between baseline and time 1. Third, the initiation of the VAN activities, including survey respondents’ engagement in the VAN platform development prior to the baseline survey, may have affected baseline measures, and therefore, this should be kept in mind when interpreting the survey results. Scorecard The scorecard has been developed to pull together a snapshot view of key KPIs, including a series of KPIs measured by the survey. Since the VAN is meant to bring together people, processes, policy, and technology to transform the way the FP community makes supply chain decisions, the scorecard is designed to measure progress across those conceptual framework elements, as well as the value propositions related to efficiency, effectiveness, scale, and cost. It offers quick and easy analysis of return on investment by key stakeholders. The scorecard frame below (Figure 9) includes a column on the left that links the relevant scorecard subjective KPIs with the relevant survey question. The column on the right includes the calculations that were used to generate the results. - 23 - Figure 9. Scorecard and survey alignment and calculations. - 24 - - 25 -

View the publication