2019 VAN Scorecard and Narrative

Publication date: 2019

VAN Scorecard and Narrative 2019 - 2 - Scorecard progress against efficiency, effectiveness, scale, and cost Overview The expectation is that putting in place a VAN with established governance (people), processes, policies, and technology will ultimately lead to more efficient and effective collaborative supply chain management, allowing scale-up coverage to more countries and products without costing more. In August 2018, the Super User Group and Steering Committee agreed on a set of specific KPIs to measure efficiency, effectiveness, scale, and cost improvements (see Figure 7). Demonstrating the magnitude of these improvements is critical to establishing the value proposition of the VAN. Two different measurement methodologies were also approved. First, the KPIs include objective, quantitative indicators that will be measured by the VAN platform. Second, there are also subjective indicators that are measured by a longitudinal evaluation to show change over time. The survey is, and will continue to be, administered in alignment with key milestones throughout the course of initial VAN implementation. Survey questions have been structured to gain insight into key stakeholders’ experience with the VAN, including both the technology system and data, and the piloted collaborative supply chain processes. The scorecard has been developed to pull together a snapshot view of key KPIs across both measurement methodologies, and other relevant statistics. Since the VAN is meant to bring together people, processes, policy, and technology to transform the way the FP community makes supply chain decisions, the scorecard is designed to measure progress across those conceptual framework elements, as well as the value propositions related to efficiency, effectiveness, scale, and cost. It offers quick and easy analysis of return on investment by key stakeholders. For more information on the scorecard methodology, see Appendix A. This section uses existing data to analyze scorecard progress (see Figure 8). The data time frame for time 1 is December 2018 for the KPIs and February 2019 for the other statistics. Certain KPIs and statistics are grayed out for time 1, given that the VAN platform was only launched in live production mode on January 22, 2019. As described above, the longitudinal evaluation survey is administered around key milestones and gathers data related to the subjective KPIs. The available data for phase 1 was the survey administered in December 2018. Many of the scorecard survey indicators are grayed out to clarify that the platform was not live, and respondents could not yet reliably assess the results. The Process Improvement and System Usability percentages are shared, however, because respondents were able to assess this based on their User Acceptance Testing. The next survey is scheduled for several months after the launch of the live platform and training to allow users to have enough comparison to report in responses. As for the objective KPIs in the scorecard, these are measured by the platform, and since the platform just went live in January, there is not yet enough data to measure. They will be included in the next scorecard review. Overall, the inefficiency and ineffectiveness pain felt by those using current processes for collaborative FP supply chain processes has reduced from 5.9 to 4.4 on a 10-point pain scale. Since this is only one data point in time, it will be important to continue to monitor the trend and see if it continues a downward trajectory over time. In terms of people and policy, the scorecard offers numerical support to the return on investment described in the foundational impact above. The scorecard highlights the breadth of community involvement in phase 1. Buy-in has been achieved on all key decisions across six Steering Committee member-organizations, four of which vote, and 16 Task Force member-organizations. As of February 2019, there are 21 active, individual VAN members who have signed in, accepted the TOU, and started using the platform. As for efficiency, effectiveness, cost, and scale associated with the process side, available data have been included in the scorecard and are further explained below. Efficiency Under efficiency, most scorecard KPIs could not be measured yet. The scorecard does include baseline and time 1 responses for the Process Improvement indicator, however. Of those that responded based on their User Acceptance Testing experience, 75 percent indicated that it was "somewhat easy" or "very easy" to access the supply chain reports and data analyses needed for review with the FP community, versus only 19 percent at baseline. In comparing average ratings on the difficulty versus ease of access to supply chain reports and data analysis for the respondents, the average ratings improved for all actors, per Table 1 below. VAN Scorecard and Narrative 2019 - 3 - Table 1. Average ratings on the difficulty versus ease of access to supply chain reports for respondents. Actor/Role Baseline (n=6) Pilot 1 (n=4) Control Tower Data Analyst 2.0 4.0 Control Tower Administrator 2.0 3.5 Global Procurer 1.3 4.0 In the future, the expectation is to see further positive statistics in the efficiency section of the scorecard, due to key efficiency gains established in phase 1: • Process Improvement. Harmonized Product Master between UNFPA and USAID that includes known Global Trade Item Number attributes and United Nations Standard Products and Services Codes. • Time Savings and Visibility. Established single system for tracking orders, shipments, inventory, and supply plans across donors. • Visibility. Expanded breadth of data types able to be loaded, including the following: o Any country that is using PipeLine for supply planning can load planning data into the system via a drag and-drop interface. o Any manufacturer can provide updates directly to the system via a drag-and-drop interface. • Time Savings, Visibility, and Process Improvement. Established Global Data Synchronization Network (GDSN) subscription, allowing the VAN to pull master data directly from the GDSN data pool for subscribed manufacturers in the future. • Time Savings. Tested integration automation on pulling stock/inventory information directly from country electronic logistics management systems. • Process Improvement. Demonstrated ability to merge CSP and CARhs into one Control Tower Logistics Management Unit, thereby leading to future time savings. These are all critical foundational efficiency gains for the future. Since the platform just launched, and process change design is still underway, users are currently working in the old systems, while trying out and developing the new processes. The result is more work now for a reduction in the future. Effectiveness Under effectiveness, it is important to look at current users’ future expectations of the VAN, based on their involvement to date. Eighty-five percent of respondents expect in the future that the VAN will allow them to make more timely and specific supply chain recommendations and decisions regarding Nigeria and Malawi compared to before. This indicator measures better coordination and will be important to follow moving forward to see if that expectation grows as data access and visibility increase. System usability and workflow improvements are important when looking at return on investment. It is expected that improved usability and work status transparency will ultimately have a hand in improving supply chain actions and access to product. Twelve respondents were able to respond based on their experience in staging and User Acceptance Testing. Forty-two percent of respondents currently “agree” or “strongly agree” that the status and progress of the collaborative FP supply planning process is transparent and visible at all times, compared to 27 percent at baseline. In comparing average ratings on status transparency and visibility for the respondents, with both a baseline and time 1 numerical value (see Table 2), average ratings improved for all actors. VAN Scorecard and Narrative 2019 - 4 - Table 2. Average ratings on status transparency and visibility for respondents. Actor/Role Baseline (n=15) Pilot 1 (n=12) Control Tower Data Analyst 4.0 4.5 Control Tower Administrator 1.0 4.5 Global Procurer 2.0 3.0 Control Tower Planner 2.1 2.8 The expectation is that the workflow ticketing module, plus the alerts and notifications on performance issues and changes, will make the system more user-friendly than current processes. This indicator is moving in the right direction and should increase as respondents interact on the live platform. Scale Having more efficient and effective collaborative supply chain processes is meant to allow the community to engage in collaborative supply chain management across more countries and products than was previously possible without adding additional costs. In the longitudinal evaluation, 63 percent of those who responded agreed that that is “likely” or “very likely.” That is more than the majority, but there is hesitancy. Twenty-five percent are still unsure on the two scale indicators because they are waiting to see how processes change with the technology, which makes sense given that the platform was not yet live when the survey was administered. This indicator is expected to increase once respondents are on the live platform. In the meantime, what is agreed is that more visibility and access to data are necessary for any movement on scale and reduction in the inefficiencies of the old processes. Under phase 1, the VAN exceeded visibility expectations around the breadth of data already loaded and visible on the platform, with 105 countries’ order and shipment data loaded, and 37 countries’ PPMR inventory loaded. In addition, order and shipment data have been loaded for all USAID and UNFPA FP product families, except condoms. The increased visibility, coupled with the process improvements and system usability, is expected to lead to increased positive statistical responses in the future. Cost The Management Unit has made a concerted effort to gather quantitative understating of baseline soft and hard costs related to RHI, PPMR, CSP, and CARhs for comparison over time to document cost savings. The challenge has been gaining access to the data, given the number of years and changes in personnel since the tools and groups were established. Enough of the CSP and CARhs soft costs have been gathered to compare cost savings as they merge moving forward. Cost comparisons due to process change will start to be measured in the future, once the two groups officially merge. In the meantime, it is useful to look to the survey and the expectations of users. Ninety-seven percent of respondents "agree" or "strongly agree" that current systems (CSP Online, RHI, PPMR) could be retired if the VAN covered the required countries and products. The ability to replace all these separate tools, with their separate maintenance and management costs, is an expected cost savings. The CSP Online Tool was already retired in phase 1. Phase 1 also established the feasibility to replace RHI, both in terms of secured data access via the VAN platform and public access via the Reproductive Health Data Visualization (RH Viz) dashboards, assuming the condom product master issues are finalized with UNFPA. The VAN initiative will continue to explore how to document these cost savings. https://www.rhsupplies.org/activities-resources/tools/reproductive-health-supplies-visualizer-rh-viz/ 2019_VAN_Scorecard.pdf 2019_VAN_Scorecard_Narrative_PDF.pdf Scorecard progress against efficiency, effectiveness, scale, and cost Overview Efficiency Effectiveness Scale Cost

View the publication