VAN Scorecard Report - 2023

Publication date: 2024

VAN Scorecard Report 2023 Tim e 1 Tim e 2 Tim e 3 Tim e 4 Tim e 5 Tim e 6 Ba se lin e Average rating on the pain scale to rate experience using the current systems and processes for collaborative FP supply chain mgmt decisions SU  ALL 5.9 4.4 4.1 3.9 3.2 3.2 3.5 # members organizations in the cross-organizational Steering Committee that meets regularly to advise on rollout - 6 6 6 8 8 10 # of multisectoral task forces (by topic area) identified and formed to advance VAN activities - 4 4 4 4 4 4 # entities participating in VAN steering committee and task forces - 16 16 18 18 25 36 # countries involved - 2 2 4 34 37 36 # manufacturers involved - 4 4 4 4 4 5 EF FI CI EN C Y % of respondents reporting “a lot less” or “less” time spent on triangulating data each month to make collaborative supply chain mgmt decisions B P  ALL - - - - - 61% 63% % of respondents reporting “a lot less” or “less” time spent on communication to review data and make supply chain decisions together B P  ALL - - - - - 53% 73% % of “past due” orders that do not have associated shipment records V  ALL - 1% 6% 5% 4% 4% % of respondents reporting “a lot less” or “less” time spent on triangulating data each month to make collaborative supply chain mgmt decisions for Nigeria and Malawi TS - 0% 21% 44% 53% 82% 79% % of respondents reporting a “a lot less” or “less” time spent on communication to review data and make supply chain decisions together for Nigeria and Malawi TS - 0% 38% 53% 41% 75% 63% % of respondents indicating that it was “somewhat easy” or “very easy” to access the supply chain reports and data analyses needed for review with the FP community PI  ALL 19% 75% 63% 47% 76% 60% 52% EF FE C TI V EN ES S % of respondents who report that the data they currently have allow them to make timely and specific supply chain recommendations and decisions B P  ALL - - - - - 64% 69% % respondents who expect in the future that the VAN will allow them to make more timely and specific supply chain recommendations and decisions compared to before B P  ALL - - - - - 78% 81% % of respondents who “agree” or “strongly agree” that the status and progress of the collaborative FP supply planning process is transparent and visible at all times B P  ALL - - - - - 81% 89% % of targeted countries providing complete supply plans at least once per quarter BC  ALL - 67% 100% 85% 85% 84% % of targeted countries providing complete inventory updates BC  ALL - 95% 95% 90% 95% 87% % of supply plans with a projected MOS below Min alert as of the end of the quarter DM  ALL - - 16% 42% 60% 46% % of Action Request tickets resolved in line with the original request for the year-to-date BC  ALL - 45% 50% 95% 90% 78% % of respondents who report that the data they currently have allow them to make timely and specific supply chain recommendations/decisions regarding Nigeria and Malawi* DM 41% 0% 44% 39% 87% 70% 73% % of respondents who expect in the future that the VAN will allow them to make more timely and specific supply chain recommendations and decisions* regarding Nigeria and Malawi compared to before DM - 85% 77% 87% 94% 82% 80% % of respondents who “agree” or “strongly agree” that the status and progress of the collaborative FP supply planning process are transparent and visible at all times regarding Nigeria and Malawi SU 27% 42% 69% 84% 83% 83% 88% % of respondents who “agree” or “strongly agree” that they are able to reliably anticipate expected arrival dates of FP commodities BC  ALL 24% 33% 43% 44% 65% 63% 62% S C A LE % respondents who report that it is “likely” or “very likely” that with the VAN processes and systems, they will be able to cover more countries than currently possible without increasing total work hours SC  ALL - 63% 75% 80% 88% 87% 67% % respondents who report that it is “likely” or “very likely” that with the VAN processes and systems, they will be able to cover more products than is currently possible without increasing total work hours SP  ALL - 63% 75% 80% 88% 80% 67% CO S T CSP Online Tool RETIRED 2019 PPMR Tool RETIRED 2022 RHI Tool RETIRED 2021 PPT and Pipeline NOT RETIRED** # versions of official Terms of Use (effective dates) - 1 1 2 4 4 4 # of logins (month of survey) - 342 570 814 979 1159 # official VAN users (accepted the TOU) - 21 58 124 463 592 760 # VAN member organizations (accepted the TOU) - 15 95 99 123 Countries with Supply Data Plan 24 (SCOPE: 24) Methods*** 20 (SCOPE 9) Products*** 79 (SCOPE 25) Countries with orders and shipment data**** 148 (SCOPE 136) Countries with inventory data**** 35 (SCOPE 46) O VE R A LL PE O PL E PO LI CY T EC H N O LO G Y PR O CE SS NOT LIVE NOT LIVE NOT LIVE NOT LIVE NOT LIVE NOT LIVE NOT LIVE Baseline and Time 1, 2, 3, 4, 5 and 6 are defined in the VAN Scorecard Narrative. * Aggregate of the following supply chain decisions and recommendations: adjusting orders and shipment timing to reduce stock imbalances; funding proposals to better align demand with limited resources; adjusting supply plans to avoid future shortages, stockouts, and overstocks; and planning production and shipment schedules. ** It is no longer necessary for PipeLine and PPT to retire. The VAN is currently integrated with PipeLine and QAT, countries can choose to use the tool that works best for their processes. The PPT is linked to UNFPA ERP transition. *** The count of method and product coverage beyond the total universe has to do with approved requests for product expansion. **** The total universe has been updated to reflect the complete coverage of active countries in the RHI dataset as of 2017. Since 2017, the highest coverage has been 136 countries. The complete set of 136 was transferred to RH Viz, and now 148 countries represent the complete coverage. ***** The total universe has been updated to reflect the coverage of 46 countries proposed in the 2020 business case. B BC C DM P PI BASIC BUSINESS CASE COST DECISION-MAKING PREMIUM PROCESS IMPROVEMENT The scorecard data collection methodology changed in Time 5 (see narrative) and now includes over 100 respondents from across all VAN member countries. This symbol is used in the Overall and Process category, and represents indicators related to all VAN member countries. Where there is no symbol, the results represent only Nigeria and Malawi, in line with the original methodology. SCALE COUNTRIES SCALE PRODUCTS SYSTEM USABILITY TIME SAVINGS VISIBILITY SC SP SU TS V  ALL 2VAN SCORECARD 2023 Scorecard progress against efficiency, effectiveness, scale, and cost Overview 1. Historically, the longitudinal survey is conducted with a small group of respondents (n=19), querying them on an ongoing basis over time in order to accurately compare trends. However, VAN membership and use has grown exponentially over the years, now covering close to 100 organizations and hundreds of individual users. In 2022, the VAN Manage- ment Unit modified the survey methodology to continue the series of repetitive cross-sectional studies with the initial three groups surveyed before, and added two new question- naires to represent Basic and Premium members not surveyed before. See the methodology section for more details. 2. The data time frame for Time 1 is December 2018 for the survey and platform KPIs and February 2019 for the other indicators. For Time 2, the data time frame is August 2019 for the survey KPIs and the fourth quarter of 2019 for the platform KPIs and other indicators. For Time 3, the data time frame is January 2020 for the survey indicators and the first quarter of 2020 for the platform and other indicators. For Time 4, the data time frame for the survey indicators is April – May 2021, and October 2021 for the platform and other indicators. For Time 5, the data time frame is July 2022 for the survey indicators, and September 2022 for the platform and other indicators. For Time 6, the data time frame for the survey indicators is June – July 2023, and September 2023 for the platform and other indicators. The VAN Scorecard Narrative uses existing data to analyze scorecard progress (see 2023 VAN Scorecard above). The scorecard is designed to pull together a snapshot view of key indicators across both the objective key performance indicators (KPIs) measured by the platform and the subjective KPIs measured by the longitudinal evaluation survey, as well as other relevant statistics.1 Six timeframes are included in this version of the scorecard: 2018, 2019, 2020, 2021, 2022, and 2023.2 As noted in previous years, certain KPIs and statistics are grayed out for the “Time 1” column, given that the VAN platform was only launched in live production mode on January 22, 2019. In 2021, as we decommissioned the Procurement Planning and Monitoring Report (PPMR), we began the process of transitioning the 35 PPMR countries to the VAN, thus expanding the pool of VAN users. To capture the viewpoints of all VAN users more accurately, the Time 5 survey was administered to the three original groups, Control Tower Members, Early Adopters, and Procurers, as well as two new groups: Basic Members and Premium Members. A new skip pattern was also introduced in Time 5 to provide survey respondents with the opportunity to indicate “I don’t support Nigeria and Malawi” and “I don’t support countries other than Nigeria and Malawi” where applicable. Time 5 provided baseline measurements for the Basic Members and Premium Members groups and Time 6 provides a second measurement for comparison. Between the Baseline survey in 2018 and survey Time 5 in 2022, consistent improvements have been seen in the inefficiency and ineffectiveness pain felt by all respondents from an initial average rating of 5.9 in the Baseline to 3.2 in Time 5. In Time 6, the average pain scale rating slightly increased from 3.2 to 3.5. While an average rating of 3.5 on a scale of 1 to 10 is still low, this slight increase in pain felt across respondents may reflect the increases in new members and users during the Time 6 timeframe. New members and users need time to learn the VAN and incorporate it into their own internal processes to reduce pain from inefficiency and ineffectiveness in those existing processes. Notable feedback from Time 6 respondents are quoted below (with the respondent’s security role listed in parenthesis after the quote). These quotes show the value users continue to find in the VAN, and reflect challenges that may explain the increase in pain scale rating: “The tool allows me to have visibility of procurement which is not only UNFPA procurement and is pretty user friendly. The pain is that you require to dedicate time (like with everything else) to well understand all the functionalities.”  PROCURER “…The VAN has opened my eyes that technology can make your life so easy. For example, before the VAN there were so many challenges as regarding supply chain issues, data and finances but when [country redacted] adopted the VAN as better solution now things are easy because all the information can be found at one place which is the VAN. So GFPVAN is the answer for all supply chain issues.”  PLANNER “Using the VAN was a positive experience with regards to learning about a new platform and features and the training was fun. As far as frustrations are concerned, there needs to be a bit of improvement to the user interface friendliness of the supply planning section.”  BASIC “VAN allows us to better track our deliveries and plan our consumption. It also allows us to monitor the availability of products on a national level.”  PREMIUM, FRENCH RESPONSE TRANSLATED TO ENGLISH 3VAN SCORECARD 2023 People and Policy In terms of people and policy, the scorecard reflects the successful and wide-ranging scope of community involvement in VAN governance and operations. The number of organizations participating in the VAN Steering Committee for governance purposes increased from 8 to 10 with the additions of the West Africa Health Organization (WAHO)/KfW as an Investor and Shaping Equitable Market Access (SEMA) Reproductive Health as a Network Player. The additions of WAHO/KfW and SEMA Reproductive Health on the Steering Committee supports the VAN’s goal of putting countries at the center of our work and ensuring country voices are part of the VAN’s governance and strategic decisions, as well as contributing to the longer-term sustainability of the VAN through an expanded donor base. The number of entities participating in VAN task force meetings also expanded over the last year from 25 to 36, which reflects an important update to include entities that joined as Premium and Premium+ members from countries. The count includes members of the four original task forces—Data Sharing, Data Management, Tech Management, and Super User—as well as the Consensus Planning Group (CPG) members and involved manufacturers. Official VAN membership and usage continue to grow. Between Time 5 and Time 6, the number of member organizations who accepted the terms of use (TOU) grew from 99 to 123, respectively. This reflects a significant amount of work done over the last year to onboard many new organizations, including social marketing and local non-governmental organizations, who were invited to join by the Ministries of Health in countries that transitioned from Basic to Premium membership, as well as Pfizer, a prominent pharmaceutical manufacturer. Similarly, the number of individual users rose from 592 in Time 5 to 760 in Time 6, which reflects a general trend across countries to expand the number of stakeholders accessing the VAN as they begin to better understand the value of its use. The number of government stakeholders accessing the VAN grew, and continues to be the largest percent of logins at the country level (see line graph on the right). Logins at the country level grew in all partner categories, reflecting the increase in member organizations and users. In addition, it also reflects the value of the support provided by the VAN Management and Control Tower Analysts to conduct refresher trainings to countries looking to expand the number of users and the large focus on integrating the VAN into the West Africa Early Warning System. Furthermore, the average number of logins has also continued to increase from 979 in Time 5 to 1159 in Time 6. The continued rise in members and users can in part be attributed to the increase in workshops to train, onboard, and enhance use of the VAN for Basic and Premium users. 0 500 1000 1500 2000 2500 3000 3500 4000 1003 1580 1673 1154 586 273 876 462 222 598 224 185 GOVERNMENT IMPLEMENTING PARTNER NGO COUNTRY DONOR 2021 2022 2023 VAN LOGINS BY YEAR 4VAN SCORECARD 2023 Processes EFFICIENCY In 2022, Time 5 established baseline measurements for the new Basic Members and Premium Members surveys. The Time 6 results show an increased efficiency through less time spent triangulating data and communicating to review data for collaborative decision-making. For Basic and Premium survey respondents, 63 percent reported less time triangulating data on a monthly basis in Time 6 compared to 61 percent in Time 5, and 73 percent reported spending less time communicating to review data in Time 6 compared to 53 percent in Time 5, a large change compared to other changes seen between Time 5 and Time 6. These figures show that as new members became more comfortable with the VAN and incorporated it into their processes between Time 5 and 6, their ability to use the VAN efficiently increased. The following feedback brings these numbers to life by contextualizing how Time 6 respondents are using the VAN to increase efficiency: “With the gradual synchronization of particular shipment, data quality checks, triangulation is reduced. It is making visibility easier to capture and as such, reduced time taken to access various data from different sources.”  PLANNER “Life has been made easy and I have more time conducting other duties since all the updates are already discussed by the Ministry and partners involved before done. There are no back- to-back collaborations, the country quickly makes supply chain decision from VAN following the teams’ technical know-hows instilled.”  PLANNER “It combines multiple source of information hence time is saved from accessing the needed information from one source.”  BASIC “The VAN will generate the data items needed during quantification which easily can be extracted from VAN with a view to make a reasonable decision on how much is needed.”  BASIC For long-time members Malawi and Nigeria, 79 percent of respondents reported less time triangulating data on a monthly basis in Time 6 compared to 82 percent in Time 5, and 63 percent of respondents reported spending less time communicating to review data in Time 6 compared to 75 percent in Time 5. This drop in Time 6 rating may be attributed to the time and effort given to improving data quality to support the ongoing integration between the VAN and Malawi’s electronic Logistics Management Information System (eLMIS), which brings in data from lower supply chain levels to the VAN’s analytics and supply plan workflows (making it visible to Malawi users for the first time). While this work takes time, it is overall helping data accuracy for more effective decision-making. Another factor may be the survey respondent’s role in the chain of communication regarding data, meaning some respondents may play a larger role in the data evaluation than others. The wider group of respondents continued to indicate slightly more difficulty accessing the supply chain reports and data analyses in Time 6. The percent of respondents who felt it was easy to access key reports and analyses to enable collaboration decreased from 60 percent in Time 5 to 52 percent in Time 6. A contributing factor that may be impacting perceived access to reports and analyses is the number of new users who have joined as a result of transitions for countries from Basic Membership to Premium Membership in the VAN, which requires additional training and time to learn how to access supply chain reports and data analyses in the platform. Another measure of efficiency that impacts the experience of country users is the percentage of orders which are past their estimated shipment date, but which do not have a shipment record in the VAN. The percentage of past due orders without shipment records held steady at 4 percent of past due orders not having an associated shipment record in Time 6 and Time 5. EFFECTIVENESS Most of the effectiveness metrics have continued to improve since the baseline and exciting improvements have been seen between Times 5 and 6 for the new measures related to Basic and Premium members. The ratings from Basic and Premium respondents on currently making timely and specific supply chain recommendations has improved from 64 percent in Time 5 to 69 percent in Time 6. For Nigeria and Malawi, respondents’ ratings on their ability to currently make timely and specific supply chain recommendations have also slightly increased from 70 percent in Time 5 to 73 percent in Time 6. Although the percent of respondents who feel they are able to reliably anticipate the expected arrival dates for FP commodities has dropped slightly from 63 percent in Time 5 versus 62 percent in Time 6, this metric has seen an overall improvement since the baseline where only 24 percent of respondents felt they were able to reliably anticipate arrival dates. Recent decreases may also reflect some of the changes in source systems for VAN shipments (such as UNFPA’s ongoing ERP replacement) that impact arrival date accuracy. 5VAN SCORECARD 2023 Respondents remain positive about the VAN’s impact on their ability to anticipate problems and make changes to their plans. Time 6 respondents had the following additional comments regarding this indicator: “The added visibility of procurer orders UNFPA, GHSC-PSM, WAHO and IPPF has extended the ability of the users to better align, predict and manage commodity procurements planning.”  ANALYST “Coordination existing in [country redacted] through VAN meetings done biweekly and RHCS quarterly meetings aids in clearing issues affecting [country redacted] FP commodity Supply Plan.”  PLANNER “The use of VAN allows for the anticipation of order delays, deliveries, and the evaluation of gaps. The availability of information on these questions allows for the readjustment and development of alternative solutions (redeployment of stock, loans to other countries, etc.).”  BASIC, FRENCH RESPONSE TRANSLATED TO ENGLISH “The VAN facilitates communcation for all actors (funders, suppliers, manufacturers, GAS agents). Everyone has information in real time and this allows for the anticipation of failures (overstocking and risks of shortages).”  BASIC, FRENCH RESPONSE TRANSLATED TO ENGLISH “This is pretty obvious as the VAN even guides you and makes suggestions on adjusting orders and delaying shipments or when to fill certain gaps in commodities.”  PREMIUM “The supply plans are developed based on the estimated consumption. Throughout the period, if we notice a problem, the supply plan can simply be updated to avoid additional costs (overstocking, shortages, lapses, etc.).”  PREMIUM, FRENCH RESPONSE TRANSLATED TO ENGLISH Regarding future expectations of the VAN, the ratings from Basic and Premium respondents on making more timely and specific supply chain recommendations and decisions in the future increased from 78 percent in Time 5 to 81 percent in Time 6. The percent of respondents who felt they will be able to make more timely and specific supply chain recommendations and decisions for Nigeria and Malawi in the future slightly decreased from 82 percent in Time 5 to 80 percent in Time 6. With increased data to review, given the integration of Malawi’s eLMIS with the VAN, and new users who are new to supply plan collaboration with the Ministry, this small decrease in expectations for the future is not surprising. More than ¾ of those surveyed in Malawi and Nigeria still expect to make more timely and specific recommendations and decisions in the future, even despite these changes. The Time 6 respondents who do expect increased decision-making capability in the future shared important insights in their qualitative responses: “The VAN gets better everyday as earlier noted and I am optimistic that with what we have already been able to achieve so far we can only make collaborative supply planning much easier for all VAN users across the globe.”  ANALYST “I expect that with the new demand planning module, the process of generating a funding gap analysis for countries will be more streamlined and easier to share with countries given that everything will be automatically calculated in the VAN.”  ANALYST “It is expected that VAN will provide the information on stock movement which can be used to determine stock levels with a view to avoid stockouts.”  BASIC Respondents continue to find positive transparency in VAN supply planning processes, with Time 6 showing the highest percentage thus far at 88 percent for Nigeria and Malawi respondents and 89 percent for Basic and Premium respondents. This positive perception will continue to feed into planning for the long-term sustainability of the platform and homing in on the networking and collaborative decision-making capabilities within such a transparent data management space. The metric related to supply plans remained mostly consistent over the last year with the percent of targeted countries providing complete supply plans slightly dropping to 84 percent in Time 6 from 85 percent in Time 5. The percent of targeted countries providing complete inventory updates saw a drop from 95 percent in Time 5 to 87 percent in Time 6. This drop is explained by the coup in Niger, which disrupted that country’s reporting due to staff turnover at reporting organizations, and communication disruptions. We anticipate positive trends for these indicators in the future as onboarding continues and strong supportive relationships are established with the Analyst Pool. 6VAN SCORECARD 2023 SCALE A key component of the VAN Business Case is the belief that efficient, effective supply chain processes will improve coordinated supply chain management across more countries, more products, and more actors without additional costs. Bringing this coordination to scale requires buy-in from everyone involved and consistent engagement with the reproductive health community. The continued scaling of the VAN platform as a data aggregator is evident in the technology indicators at the bottom of the scorecard. Twenty-four country supply plans are now formatted, mapped, and uploaded to the VAN. Twenty contraceptive methods and 79 products are now covered in the VAN, far exceeding the original agreed scope. The VAN is bringing in more products and actors without additional costs. The other two scale indicators only appear in the survey given to control tower members (which primarily includes the analyst pool), and are meant to be a measure of their confidence in the VAN’s ability to expand and cover additional countries and products. Sixty-seven percent of respondents agree that the VAN will be able to cover more countries in the future and 67 percent of respondents agree that the VAN will be able to cover more products in the future. This is a drop from Time 5 and likely reflects the analysts’ understanding that the current focus is on proving the wider objectives of the VAN fully with family planning products and with an active group of countries, rather than focusing on scaling out to other health areas and new countries. It also likely reflects the analysts’ growing understanding of the complexity around product and country expansion. It is important to note that this question is not asked of the membership at large, so responses may not reflect the views of all VAN members. COST A key value-added benefit of the VAN is the potential to merge multiple contraceptive supply chain management platforms into one globally accessible technology, thereby reducing operating and training costs and increasing efficiencies in decision-making and troubleshooting. In 2021, the PPMR was officially retired, and the 35 member countries were onboarded to the VAN, leaving PipeLine as the final tool to be considered for retirement. Since exploring the possibility for PipeLine retirement, the community has recognized that many countries continue to use PipeLine as their supply planning tool and may choose to do so in the future. For those who want to change, a tool called the Quantification Analysis Tool (QAT) has been rolled out by the Global Health Supply Chain – Procurement and Supply Management project. In the spirit of merging with existing platforms, the VAN currently has a direct integration with QAT and accepts exports from PipeLine so that countries can work in either tool and have their work connected to the VAN. A key benefit of the VAN is cost saving and cost mitigation related to improved supply chain management. The breadth of data and health system strengthening services accessible to VAN members has fostered more efficient reproductive health supply chain management at global and regional levels. Between 2020 and 2023, the VAN helped procurers mobilize more than $109 million in new orders for member countries, avoided waste by supporting $5.39 million worth of cancelled and transferred orders, and helped expedite 98 orders to mitigate supply shortages. VAN data and services have also proven critical in addressing country procurement gaps and rationalizing the allocation of support depending on need and funding options. In 2021 (infographic), 2022 (infographic), and 2023 (infographic) VAN Control Tower Analysts worked with countries and procurers to use the VAN to identify gaps in funding for FP commodities and more targeted funding allocation for procurement to cover the critical gaps. The data analyses and recommendations have become a valuable asset to the global community over the last few years, and we expect these practices to continue growing and expanding as the VAN expands. A growing percent of network members are starting to see enough value in the VAN’s cost benefits that they are willing to buy-in to support the sustainability of the VAN. The goal for the timeframe between October 2022 and September 2023 was that 18 percent of the VAN costs would be covered by membership subscription revenue that is separate from the initial four investor contributions. The goal was surpassed and 25 percent membership subscription revenue was received against the October 2022–September 2023 expenses. Key to this success was proving relevance and value to a growing number of players across the supply chain ecosystem. Children’s Investment Fund Foundation (CIFF) was welcomed in 2023 as the first donor in the community to join the VAN as a paying Donor Viewer member, rather than the larger Investor role. This membership allows them to view order & shipment, inventory, and supply plan data across countries and access dashboards and prepared reports in Analytics. SEMA Reproductive Health also officially joined the VAN in 2023 as a paying Network Player member given their intention at the time to leverage the VAN to carry out part of their market functions. Further validating the VAN’s cost benefit and value to countries, WAHO and the German Development Bank KfW became the VAN’s newest Investor, supporting more Economic Community of West African States (ECOWAS) countries to buy into VAN membership. Global entities like these are crucial to engage and support countries in their journey to increased ownership and stewardship. In addition to these global entities, 6 country governments paid their first annual membership dues or have had their annual membership dues paid on their behalf by a country partner. https://www.rhsupplies.org/uploads/tx_rhscpublications/CPG_-_2021_in_review.pdf https://www.rhsupplies.org/uploads/tx_rhscpublications/CPG_-_2022.pdf https://www.rhsupplies.org/uploads/tx_rhscpublications/CPG_-_year_in_review_2023.pdf

View the publication