Monitoring and Evaluation Indicators for Assessing Logistics Systems Performance

Publication date: 2006

MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE MAY 2006 This publication was produced for review by the United States Agency for International Development. It was prepared by the DELIVER project. MONITORINGAND EVALUATION INDICATORS FORASSESSING LOGISTICS SYSTEMS PERFORMANCE The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DELIVER DELIVER, a six-year worldwide technical assistance support contract, is funded by the U.S. Agency for International Development (USAID). Implemented by John Snow, Inc. (JSI) (contract no. HRN-C-00-00-00010-00) and subcontractors (Manoff Group, Program for Appropriate Technology in Health [PATH], and Crown Agents Consultancy, Inc.), DELIVER strengthens the supply chains of health and family planning programs in developing countries to ensure the availability of critical health products for customers. DELIVER also provides technical management of USAID’s central contraceptive management information system. Recommended Citation DELIVER. 2006. Monitoring and Evaluation Indicators for Assessing Logistics Systems Performance. Arlington, Va.: DELIVER, for the U.S. Agency for International Development. Abstract Quality monitoring is an integral part of any health logistics system, as it can describe the performance of the logistics system, evaluate strengths and weaknesses, and allow for accountability and advocacy. This document describes several indicators that are useful tools in monitoring and evaluating a logistics system for reproductive health commodities. These indicators can be used to measure the availability of commodities in a health facility, the performance of the logistics system, and, ultimately, commodity security. DELIVER John Snow, Inc. 1616 North Fort Myer Drive, 11th Floor Arlington, VA 22209 USA Phone: 703-528-7474 Fax: 703-528-7480 Email: deliver_project@jsi.com Internet: www.deliver.jsi.com CONTENTS INTRODUCTION . 1 INDICATORS FOR STOCk AVAILAbILITY AND LOGISTIC SYSTEM PERFORMANCE . 3 INDICATOR: Accuracy of Logistics Data for Inventory Management.3 INDICATOR: Percentage of Facilities That Receive the Quantity of Products Ordered.5 INDICATOR: Percentage Difference between the Quantity of Products Ordered and the Quantity of Products Received .6 INDICATOR: Percentage of Facilities That Maintain Acceptable Storage Conditions .7 INDICATOR: Percentage of Facilities That Experienced a Stockout at Any Point during a Given Time Period .8 INDICATOR: Percentage of Facilities Whose Stock Levels Ensure Near-Term Product Availability . 10 INDICATOR: Logistics System Assessment Tool Indicator Scores . 12 INDICATORS GUIDE FOR MEASURING FORECASTING PERFORMANCE . 15 INDICATOR: Mean Absolute Percentage Error between Forecasted Consumption and Actual Consumption of a Product . 15 INDICATOR: Percentage Difference between Consumption Forecasts and Actual Consumption . 18 INDICATORS FOR MEASURING PROCUREMENT PERFORMANCE . 21 INDICATOR: Commitment by All Stakeholders to Carry Out an Established Procurement Plan for Each Product . 21 INDICATOR: Existence of an Adequate Multi-year Procurement Plan for Each Product Offered . 22 INDICATOR: Percentage of Purchase Orders Placed That Are Filled Correctly by External Supplier (Supplier Fill Rate). 25 INDICATOR: Supplier Lead-Time Variability . 26 INDICATOR: Percentage of Average International Price Paid. 27 INDICATORS FOR MEASURING WAREhOUSING AND INVENTORY MANAGEMENT PERFORMANCE . 29 INDICATOR: Order Fill Rate (or Percentage of Orders Placed That Are Filled Correctly). 29 INDICATOR: Inventory Accuracy Rate (or Accuracy of Stock balance for Inventory Management) . 31 INDICATOR: Percentage of Stock Wasted due to Expiration or Damage. 33 INDICATOR: Order Turnaround Time. 35 INDICATOR: Inventory Turnover Rate. 36 INDICATORS FOR MEASURING DISTRIbUTION PERFORMANCE. 37 INDICATOR: Ratio of Distribution Cost to Value of Commodities Distributed. 37 INDICATOR: Average Delivery Time . 39 CONTENTS iii FIGURE Figure 1. Commodity Logistics Cycle .1 TAbLE Table 1. Instrument for Measuring Adequacy of Procurement Planning . 23 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE iv INTRODUCTION Quality monitoring is an integral part of any health logistics system because it can describe the perfor­ mance of the logistics system and point to areas that need improvement. Monitoring and evaluation also allows for accountability so that all stakeholders can know the impact of their work and investments; it also allows for advocacy as it can highlight program successes and areas of greatest need. Figure 1 shows the logistics cycle, a systematic approach to describing the activities of a logistics system. Quality monitor­ ing is shown throughout the logistics cycle, which suggests that quality monitoring should be included in every step of the cycle. The following indicators are useful tools in monitoring and evaluating a logistics system for health commodities. These indicators are used to measure the availability of commodities in a health facility; the performance of the logistics system; as well as distribution, forecasting, and procurement performance. Figure 1. Commodity Logistics Cycle Qu alit y M onit oring Quali ty M onit ori ng Quality Monitoring Quality Monitoring Product selection serving Customers Inventory Management forecasting & Procurement Logistics Information Money People MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 1 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 2 INDICATORS FOR STOCk AVAILAbILITY AND LOGISTIC SYSTEM PERFORMANCE INDICATOR: Accuracy of Logistics Data for Inventory Management DEfINITION For each method, brand, or product of interest, this indicator measures the accuracy of logistics data as the percentage of discrepancy between (1) physical stock count and stock record count, and (2) stock record count and logistics management information system (LMIS) report count. Each part of the indicator is calculated as follows: 1. Accuracy in keeping stock records: 2. Accuracy in transferring information to the LMIS reporting form: stock record count – physical stock count × 100 LMIS record count – stock record count × 100physical stock count stock record count Physical stock, stock record, and LMIS report count refer to the amount of each product that is undamaged, not expired, and available for use in a service delivery facility or warehouse. Physical stock count is determined by counting the stock in the store. Stock record count is recorded on forms that facilities use to track stock balances, transactions, and adjustments over time. LMIS report count is recorded in periodic reports that summarize stock balances, transactions, and adjustments over a specified period of time. The reports are transmitted from one level of the system to another. Evaluators can report each measure of discrepancy (or agreement) by facility or in the aggregate, and they should report each product of interest. These measures may also be useful in calculating the percentage of facilities that keep accurate stock records and produce accurate reports (defined as reports showing that discrepancies for all products fall within a margin of error agreed upon by the program). DATA REquIREMENTs • Physical counts of total number of products in the facility • Recorded inventory, which can be retrieved from the stock ledger or stock cards • LMIS reports. DATA sOuRCEs • Facility survey or logistics site visits to all facilities or to a representative sample of facilities. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 3 PuRPOsE AND IssuEs This indicator measures the accuracy of data on product stock levels at various points in the logistics system. The indicator is essential because the supply chain relies completely on stock data to forecast, procure, and deliver the right quantities of products to storage and service delivery sites. It highlights the importance of data quality at every level of the system. The first part of the indicator, which uses information on stock levels on the day of the site visit, provides information on how accurately the facilities are tracking their inventories. The second part, which compares the most recent available LMIS report to the inventory record balance closest to that date, provides information on the accuracy of the data being transferred to the LMIS reporting forms. Because the second part requires reviewing historical stock records, it may be difficult for the evaluators to collect these data. This indicator can also be used to check for leakage in the system, track timeliness in updating stock records, and determine the extent to which programs complete and submit LMIS reports. Ideally, a program should not have discrepancies between the physical inventories and the two sources of stock-level data, but, in practice, evaluators should expect some errors. Acceptable levels of error will depend on conditions in each country. In general, discrepancies of more than 10 percent should cause concern and may require efforts to improve data quality. RELATED INDICATORs • Percentage of facilities that keep accurate logistics data for inventory management • Percentage of facilities that completed and submitted an LMIS report for the most recent reporting period. � MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATOR: Percentage of facilitiesThat Receive the quantity of Products Ordered DEfINITION For all products that the program is committed to supply, this indicator measures the percentage of difference between the amount ordered in the last order period (or other defined period of time) and the amount received for that period. amount received – amount ordered × 100amount ordered This indicator should be calculated for each product for which an order is placed. If a mean order fill rate figure is desired for all products, the figure should be based on the absolute values of the discrepancies calculated for each product. DATA REquIREMENTs • List of products that the program has committed to supplying or a predetermined subset of that list • Quantity of products ordered for the last order period or during a defined period of time (e.g., quarter, year) and the dates that the orders were placed for all facilities or a representative sample of facilities • Quantity of products received for the last order period or during the same defined period of time (e.g., quarter, year) • Dates that the orders were received for all facilities or a representative sample of facilities. DATA sOuRCEs • Facility survey or site visits to all facilities or to a representative sample of facilities. PuRPOsEs AND IssuEs This indicator measures the order fill rate for selected products during a defined period of time. To enable evaluators to better understand how staff are managing their stock, the indicator shows whether orders are being completely filled in a timely manner. The indicator can be calculated at each facility to identify problematic products, suppliers, or both. It can also be used to identify areas that need improvement at other levels of the logistics system where facilities determine their own order quantities, including the national level. RELATED INDICATORs • Average duration of time between the date an order was placed and when it was received • Percentage of facilities that received their last four orders according to schedule. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 5 INDICATOR: Percentage Difference between the quantity of Products Ordered and the quantity of Products Received DEfINITION For each product that the program is committed to supply, this indicator requires a calculation of the percentage difference between the quantity ordered the last time an order was placed and the amount actually received. The indicator should be calculated separately for each product for which orders are placed between levels of the logistics system or for which national procurement orders are placed. It can be calculated at the individual facility level, for different levels of the system, or for the country as a whole. In most cases, results will be aggregated across many facilities and orders, and in such cases, the indicator should be calculated using the sum of the absolute values of the order discrepancies for each product, divided by the sum of all order quantities. sum of quantities received – sum of quantities ordered × 100sum of quantities ordered DATA REquIREMENTs • List of products that the program has committed to supply or a predetermined subset of that list • Amount of products requested the last time an order was placed for all facilities and/or warehouses or a representative sample of facilities and/or warehouses, or the amount of products ordered for the most recent national procurement • Amount of products actually received in response to the last order or procurement. DATA sOuRCEs Facility survey or site visits, or procurement records for national procurement orders. PuRPOsEs AND IssuEs This indicator measures the efficiency of a supply chain in ensuring that products reach their destinations in the quantities requested. It can be calculated for the supply chain as a whole or for any level or facility that receives supplies based on an order to a higher level. The information can reveal which products are frequently under- or over-supplied and which suppliers or distributors are most or least reliable. Managers and evaluators can use this information to take corrective actions and improve supply chain efficiency. Caution should be exercised when interpreting this indicator for non-full-supply products. Many of those products are rationed because of limited resources, so it is to be expected that they would experi­ ence greater order discrepancies than full-supply products. Such discrepancies may be more due to lack of funds to procure supplies than to inefficiencies of the supply chain. RELATED INDICATORs • Percentage of all orders that are completely filled • Average duration of time between the date an order was placed and when it was received • Percentage of facilities that received their last order completely filled. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 6 INDICATOR: Percentage of facilitiesThat Maintain Acceptable storage Conditions DEfINITION This indicator measures the percentage of storage facilities that meet acceptable storage conditions. Evalu­ ators should report this indicator for each condition listed in the LIAT. no. of facilities meeting each acceptable storage condition × 100total no. of facilities visited DATA REquIREMENTs • Checklist of acceptable storage conditions • Data collected for each condition for all facilities or for a representative sample of facilities by an observer knowledgeable about storage requirements. DATA sOuRCEs Facility survey or site visits to all facilities or to a representative sample of facilities. PuRPOsE AND IssuEs This indicator measures the conditions of storage facilities compared with a list of conditions required to protect the integrity of products. Evaluators can apply the indicator at each level of the logistics system to identify facilities that need improvement. Evaluators should use the first part of the checklist found in the LIAT to assess all storage facilities (including small storage spaces at the service delivery point level). They should apply the second part of the list to larger facilities, as appropriate. RELATED INDICATOR • Percentage of facilities meeting all (or a desired percentage) of the storage conditions. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 7 INDICATOR: Percentage of facilitiesThat Experienced a stockout at Any Point during a GivenTime Period DEfINITION This indicator measures the percentage of facilities (service delivery points, warehouses) that experienced a stockout of a method, brand, or product expected to be provided or issued by that site at any time during a specified period (e.g., the past 6 or 12 months). no. of storage facilities assessed that experienced a stockout of a method/brand/product × 100total no. of facilities assessed that distribute or issue method/brand/product and that have the data available Evaluators should calculate the indicator at all (or at a sample of ) facilities that distribute or issue prod­ ucts. They should calculate the indicator separately for each product and aggregate the data to calculate the percentage of facilities that experienced a stockout of each product, at any time, during the specified period. Evaluators may use the stock status table in the LIAT to tabulate data required to measure the indicator. DATA REquIREMENTs Information on stock levels of all products of interest for the past 6 (or 12) months at all levels of the system. DATA sOuRCEs Evaluators usually need to make a facility survey or site visit at all facilities or at a representative sample. In some countries or programs, evaluators may use LMIS or supervisory records, depending on the qual­ ity of the information available. PuRPOsE AND IssuEs This indicator measures product availability (or absence) over a period of time and serves as a proxy indicator of the ability of a program to meet clients’ needs with a full range of products and services. Evaluators should use this indicator in conjunction with the stock status indicator; they should interpret it with caution because facilities can avoid stockouts by rationing supplies. Other related indicators (see below) may provide more information on overall product availability. For example, duration of stockouts may help differentiate between products out of stock for a short period of time (e.g., one or two days) and those out of stock for extended periods. Evaluators may assess reasons for stockouts to help program managers address the underlying causes for this logistics system failure. If national policy dictates that different brands of the same product cannot be used interchangeably, then evaluators should monitor brands separately. If the policy allows substitution of equivalent brands, and if providers make such substitutions in practice, then evaluators can monitor different brands as a single product. Using data for a 12-month period allows evaluators to consider seasonal variations in product use, but it may be difficult for evaluators to obtain the historical data. Calculating this indicator using data for 6 months is less cumbersome because it requires reviewing fewer reports. If evaluators rely on data from fewer than 12 months, they should investigate seasonality issues when appropriate. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 8 RELATED INDICATORs • Percentage of facilities out of any product on day of visit • Percentage of facilities fully stocked (all products) on day of visit • Mean number of products out of or in stock on day of visit • Percentage of products out of stock or not stocked out at any time during past 6 (or 12) months • Mean number of times each method was out of stock in the past 6 (or 12) months • Mean duration of stockouts. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 9 INDICATOR: Percentage of facilities Whose stock Levels Ensure Near-Term Product Availability DEfINITION This indicator measures the percentage of facilities with stock levels above the established minimum level and below the established maximum level for each full-supply method, brand, and product of interest, at a specified time (e.g., the day of the visit). no. of storage facilities that have stock levels above the established level but below the established maximum level for the product × 100total no. of facilities visited For non-full-supply products, if stock levels are below the established minimum level, evaluators should record whether an outstanding order exists for replacement stock. Evaluators can report the indicator at the facility level or can aggregate it for a sample of facilities or for the entire program. At any level, evaluators should calculate and report the indicator separately for each product of interest to ensure that each product receives an individual measure. Averaging all products for an average stock-level adequacy is not recommended because oversupply in one product can cancel out undersupply in another, falsely implying that average stock levels were adequate. DATA REquIREMENTs • Stock levels of all products of interest at a point in time (e.g., the day of the visit) • Maximum and minimum stock levels established by the program • Historical consumption or issues data for each product at each facility • Records of recent orders (for products below minimum levels). DATA sOuRCEs To assess stock levels, evaluators must often make a facility survey or site visit at all facilities or visit a representative sample. Evaluators may collect stock data by taking a physical inventory or by reviewing the stock ledger or stock cards. In some countries or programs, the LMIS or supervision records may provide usable stock-level data. The LMIS should also provide maximum and minimum stock levels with consumption data, by product. Service statistics or similar records may provide the needed data on consumption or issues if the LMIS does not. PuRPOsE AND IssuEs This indicator provides an overall measure of whether stock levels of products are adequate at a point in time. It helps reveal overstock situations that could lead to product expiration and wastage as well as low stock levels that could result in stockouts or rationing. In applying this indicator, evaluators must care­ fully evaluate facilities where stock quantities are below established minimum levels. To do so, they should determine whether a new order was placed when stock levels reached the minimum. If such an order is outstanding, they may consider stock status adequate because the order will probably arrive before the facility runs out of stock. If not, the stock status is inadequate. 10 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE Evaluators should apply the indicator to products that the program has committed to keep in full supply. Stock status at any point in time for products that are not in full supply may reflect only the length of time since the last shipment arrived rather than measure whether inventory management procedures are effective. For non-full supply products, stakeholders should agree upon appropriate parameters to use to measure near-term product availability. Ideally, evaluators will measure stock status over a period of time (see related indicator below), but, that approach is usually possible only where the LMIS is automated. RELATED INDICATORs • Percentage of time during a given period that each product of interest is adequately stocked (this indi­ cator requires an automated LMIS system or extensive review of historical stock ledgers) • Percentage of facilities with all full-supply products adequately stocked for near-term availability • Percentage of facilities that are understocked, adequately stocked, and overstocked. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 11 INDICATOR: Logistics system AssessmentTool Indicator scores DEfINITION The Logistics System Assessment Tool (LSAT) allows a comprehensive system-level assessment of the performance of a logistics system for any health program managing any health commodity. Evaluators can use the tool with the other logistics indicators in this section to completely assess a program’s ability to continually provide health commodities at service delivery points. DATA REquIREMENTs • System-level scores for each LSAT section • Detailed information about logistics system processes. DATA sOuRCEs Evaluators should complete assessments by consensus among program managers or among others with knowledge of logistics management and system operations and performance. PuRPOsE AND IssuEs If the programs to be assessed are separate and vertical, the evaluator should complete this tool separately for each program. The instrument is organized according to the components of the logistics cycle shown in the introduction: organization and staffing, logistics management information system, product selec­ tion, forecasting, obtaining supplies and procurement, inventory control, warehousing and storage, trans­ port and distribution, product use, and finance/donor coordination/commodity security planning. Each section contains a series of objective and quantifiable yes/no questions, as well as open-ended qualitative questions that explore strengths and weaknesses of the logistics system. The combination of yes/no and qualitative questions allows evaluators to use the tool for both monitor­ ing and diagnostic purposes. Evaluators can average and score quantitatively the yes/no questions in each section to assess progress and improvements in a given logistics system over time, whereas the qualitative questions can help evaluators more clearly understand the system’s strengths and weaknesses. Qualitative questions should also reveal the causes of areas of weakness and recommend potential ways to improve them. The main issue to consider when using the LSAT (from a monitoring perspective) is its potential subjec­ tivity. If evaluators use a consensus exercise to answer each question, experts may disagree about what the response should be. In general, most yes/no questions are as objective as possible; evaluators should easily find evidence to support one side or the other. In some situations, a condition may be met at some levels of the system or in some regions of the country, but not in all. In general, most questions require that the condition be met at all levels (central down to service delivery point) to receive a “yes” answer, so again most scoring should be unambiguous. Finally, the question of reliability of scores over time may be an issue, given that different facilitators or participants may be involved (though evaluators are advised to maintain group consistency to the greatest extent possible). As with the other examples, this potential problem is minimal because the yes/no questions are as clear and objective as possible, but subjectivity can never be completely eliminated. Evaluators should always consider this limitation when they interpret the LSAT’s results. The following summary is a partial listing of the information collected in each section of the instrument. 12 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE Organization and Staffing: Yes/no questions ask about the existence of a logistics management unit and its responsibilities. Qualitative questions ask about relations among key stakeholders and how that rela­ tionship affects logistics system performance. Logistics Management Information System: Yes/no questions assess the types of information collected through the LMIS, the purposes for which the information is used, and the extent to which the LMIS is automated. Qualitative questions try to find out the flow of information at various levels of the system, whether the information collected is used by program managers, how useful it is in practice, and whether reports are accurate and timely. Product Selection: Yes/no questions ask about the existence of a national essential drug list, the existence of a national drug policy document, and the basic characteristics of each. Qualitative questions inquire about the contents of the documents and probe to determine to what extent they are disseminated and applied at various levels of the system. Forecasting: Yes/no questions are used to determine what information is used in preparing forecasts and how frequently they are calculated. Qualitative questions examine the quality of forecasts, the effect of forecasts on budgeting and planning, and the capacity of in-country staff to carry out forecasts without external assistance. Obtaining Supplies/Procurement: Yes/no questions assess whether procurement plans take into account certain information items. Qualitative questions probe the methods for coordinating procurement plan­ ning in the country and whether, in general, the program procures the right amounts of the right goods. Inventory Control Procedures: Yes/no questions provide information on the use of state-of-the-art inventory control practices (e.g., first-to-expire, first-out (FEFO) management; on established maximum and minimum stock levels); and on whether stockouts occurred at any level during the past 12 months. Qualitative questions look at how well the staff applies the procedures in practice. They also identify which products are most frequently out of stock and why, how the staff handles stockouts and oversup­ plies, and the effects of stockouts on the program. Warehousing and Storage: Yes/no questions seek to determine the existence of guidelines for proper stor­ age of all products, for assurance of product quality, and for disposal of hazardous waste and damaged or expired products. Qualitative questions collect more in-depth information on areas where the staff could improve storage conditions and how the staff ensures product quality. Transport and Distribution: Yes/no questions check for a budgetary line item for various components. Qualitative questions assess whether transport is sufficient and whether the distribution system is effective. Product Use: Yes/no questions query the existence of standard treatment guidelines and procedures for monitoring prescribing practices. Qualitative questions probe to determine the extent to which standard treatment guidelines and universal safety precautions are actually implemented. Finance/Donor Coordination/Commodity Security Planning: Yes/no questions assess whether the national budget covers logistics, whether a cost-recovery system is used, and what logistics expenses are covered by donors. Qualitative questions examine whether budget allocations for logistics are sufficient to ensure product availability, examine the source of funds for the logistics budget, and examine whether donor activities and resources are adequately coordinated. Organizational Support for Logistics System: Yes/no questions assess the existence of job descriptions and characteristics of communication channels, information use, decision making, feedback, supervision, and training. Qualitative questions focus on how various organization and staffing procedures are carried out and how effective they are in practice. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 13 1� MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATORS GUIDE FOR MEASURING FORECASTING PERFORMANCE INDICATOR: Mean Absolute Percentage Error between forecasted Consumption and Actual Consumption of a Product DEfINITION This indicator measures the mean absolute percentage difference between a forecast previously made over several time periods and the actual consumption or issues data for those same time periods. This indicator is often referred to as mean absolute percentage error (MAPE). ∑APEiMAPE = n where APE = | forecast for timei – actual consumption for timei | × 100i actual consumption for timei Evaluators should calculate the indicator for each product for which a forecast was made. This indica­ tor should be used at the level where long-term procurement decisions are made—most commonly the central level—but it can also be applied to other levels of the system if forecasting has been decentralized and if facilities determine their own order quantities. DATA REquIREMENTs • Forecasts, by product, for a given time period • Actual consumption or issues data (if available), by product, for the same time period (Note: If actual consumption or issues data are not available, service statistics or other data can be substituted). DATA sOuRCEs Key informant interviews, review of logistics records, demographic surveys, and service statistics. The forecasts for the level being reviewed and the list of products should come from the government or other sources—for example, recommended orders to donors for essential drugs or other government forecasts. Evaluators may be able to obtain consumption data from a management information system operating at the central level for each product, or they may choose to substitute issues data from the central warehousing facility in lieu of actual consumption data. Evaluators may also estimate data from demographic surveys or service statistics. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 15 PuRPOsE AND IssuEs Accurate forecasting helps countries and organizations procure the right amount of each commodity, thus reducing the likelihood of wastage or shortage and increasing the likelihood of meeting client needs with available products. A point indicator for percentage difference between forecasted consumption and actual consumption is useful in evaluating only the accuracy of a forecasted point in time or for one aggregated forecasted amount (e.g., an amount for an entire year). The MAPE indicator allows the evaluator to review the suit­ ability of a forecast over several time periods. The evaluator must remember to calculate the absolute value of each percentage error term in the numera­ tor for absolute percentage error (APE). If the absolute value is not taken, then the mean percentage error calculated is likely to be small (because positive and negative terms will offset) and not useful for evaluation. The lower the MAPE value, the more accurate is the forecast. It is difficult to provide a threshold MAPE value below which a forecast is considered to be accurate. Evaluators will often compare the MAPE value for the desired forecast with the MAPE values of other forecasts to evaluate the accuracy of the forecast. If other forecasts are not available for testing, one simple method of forecasting to use for comparison purposes is taking the previous year’s actual issues for each product and adding a percentage increase or decrease for each product. This percentage increase or decrease can be applied to consumption data for the last available time period (e.g., from the previous year) and to all successive time periods throughout the forecast period. The difference between the MAPE of the simplified forecast and the MAPE of the actual forecast (i.e., forecast being tested) provides a measure of the improvement attainable through use of the actual model, assuming the MAPE of the actual forecast is lower than the MAPE of the simplified forecast. If, however, the MAPE of the actual forecast is higher than the MAPE of the simplified forecast, the accuracy of the actual forecast is less than the accuracy of the simplified forecast, and the organization would have to criti­ cally assess the reasons for the results. New forecasting approaches would likely be required. RELATED INDICATORs • Average MAPE of multiple products This indicator is an average of the MAPEs for all products (n) of interest. MAPEs are measured for each product i, and the average of all MAPEs is taken. The indicator is given as— ∑MAPEi n • Average percentage difference between forecasted consumption and actual consumption of multiple products 16 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE This indicator is an average of the percentage differences between forecasted consumption and actual consumption for all products of interest. Percentage errors are measured for each product (i), and the average of all percentage errors is taken. The indicator is given as— ∑PDi n where PD = | forecast consumption for producti – actual consumption for producti | × 100i actual consumption for producti MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 17 INDICATOR: Percentage Difference between Consumption forecasts and Actual Consumption DEfINITION For all products that the program has committed to supply, this indicator measures the percentage difference between forecasts previously made for a year (or other appropriate time period) and the actual consumption or issues data for that period. Evaluators should calculate the indicator for each product for which a forecast is made. If evaluators desire a mean forecast accuracy figure for all products, they should base it on the absolute values of the discrepancies calculated for each product. This indicator is most commonly measured annually at the central level, but it can also be applied more frequently at lower levels of the system as a measure of facilities’ capacity to determine their own order quantities. In either case, the basic formula is the same. The indicator is calculated as follows: | forecast consumption – actual consumption | × 100actual consumption DATA REquIREMENTs • List of products that the program has committed to supply • Forecasts or order requests by product for the period of interest • Actual consumption or issues data by product for the period of interest. DATA sOuRCEs Evaluators can obtain logistics data from LMIS reports, plus (at the national level) key informant inter­ views, records reviews, demographic surveys, and service statistics. National-level forecasts and the list of products should come from government or other sources—for example, Contraceptive Procurement Tables (CPTs) for contraceptives supplied by the U.S. Agency for International Development (USAID)—recommended orders to donors for essential drugs, or a govern­ ment (or other) forecast of product needs. At lower levels of the system, the forecasts would be repre­ sented by order requests to the next higher level. Evaluators may obtain consumption data from LMIS reports at any level, and at the national level, evaluators may estimate consumption from demographic surveys or service statistics. They can obtain CPTs or national-level forecasts by product through the local USAID Mission, from the USAID-funded DELIVER project, or from host-country program managers for contraceptive products that USAID supplies. At lower levels of the system, LMIS forms can be used to obtain data on quantities requested in orders and consumption or issues for each facility in the most recent order period. PuRPOsE AND IssuEs At all levels of the system, accurate forecasting helps countries and organizations order the right amount of each commodity, thereby reducing the likelihood of wastage or shortages and increasing the likelihood of meeting client needs with available products. A forecast made using past consumption data and sound forecasting methodologies should approximate actual consumption within a margin of error appropriate 18 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE for each product. Host-country stakeholders should agree on the allowable margin of error, and evaluators should interpret results in light of real-world conditions that may have been impossible to foresee. Fore­ casts are subject to uncertainty for many reasons, so some errors must always be accepted, particularly at the national level when the forecast period is long. Documenting the reasons for particularly wide discrep­ ancies (including assumptions used in preparing the forecast) helps put the results in perspective and may provide insights for improving future forecasts. This indicator also indirectly measures data quality, since an accurate forecast can result only if the data used are of good quality. RELATED INDICATORs • Mean level of forecast accuracy or discrepancy for a range of facilities, products, or both • Percentage of facilities with forecasts within 5 percent of actual consumption, by product. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 19 20 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATORS FOR MEASURING PROCUREMENT PERFORMANCE INDICATOR: Commitment by All stakeholders to Carry Out an Established Procurement Plan for Each Product DEfINITION For each product procured by a program, all stakeholders—donors, lenders, and program managers— should have committed to carry out the established procurement plan. Commitment from donors should ideally be in writing. Furthermore, all commitments should include a planned shipment schedule consis­ tent with the plan and a confirmation of the budget allocation for the product. All of these conditions should be met before evaluators score the indicator as “yes.” This is a yes/no indicator for each product (though it could be scaled if so desired). Evaluators may wish to qualitatively assess the strength of commitment to determine whether shipments are likely to occur as promised and as scheduled. Evaluators can assess this commitment through key informant interviews. If no procurement plan exists for a given product, this indicator does not apply. DATA REquIREMENTs • List of products the program has committed to supply • Procurement plans for program-wide product requirements, by product • Acceptable evidence (or lack thereof ) of commitment to procurement, including shipment schedules and documented planned budgetary allocations. DATA sOuRCEs Evaluators should conduct key informant interviews and records review of procurement planning docu­ ments at the level or levels where forecasting and quantification exercises take place. The product list, planned shipment schedules, and information about budget line items for products should also be available at the central level for each program. PuRPOsE AND IssuEs This indicator measures whether key stakeholders are committed to the procurement plan. Although logistics planners may not be able to control timing of actual product deliveries, they should at least secure commitments from appropriate stakeholders and should follow up with donors to increase the likelihood that shipments will arrive as planned. Gaining such commitment is an essential component of reproductive health product security. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 21 INDICATOR: Existence of an Adequate Multi-year Procurement Plan for Each Product Offered DEfINITION For each product that a program procures, a multi-year procurement plan prospectively ensures that the product will be in stock throughout the period and (for full-supply products) that stock quantities will not exceed established maximum levels. This is a yes/no indicator for each product. Ideally, procurement plans should cover three or more years, particularly for donor-supplied products which may have very long lead times. However, given the practical realities of synchronizing procurement plans with donor budget cycles, evaluators should score a plan that meets these criteria for at least two years as satisfactory. The indicator is scored affirmatively if columns 7 and 8 in table 1 show positive stock levels throughout the period of the plan and if stock levels for full-supply products do not exceed the established maximum level. DATA REquIREMENTs • List of products that the program has committed to supply • Procurement plan for program-wide product requirements • Program-wide stock levels for each product • Established maximum levels for each full-supply product • Existence of a forecast. DATA sOuRCEs Evaluators will conduct key informant interviews and records review. The procurement plan should be available at the level where procurement decisions are made. Evaluators can use table 1 to determine the indicator. PuRPOsEs AND IssuEs This indicator measures whether the program has adequately planned the procurement and shipping schedule for the products that it has committed to supply and for which it has forecasted needs. Given budgetary realities, program managers may not have plans for many non-full-supply products (or the plan may be inadequate to avoid stockouts or expiration); nevertheless, the indicator should apply to all products. The indicator is useful for showing policymakers and donors where shortfalls or oversupply may occur and for advocating changes to prevent such supply imbalances. It can also be used to check whether procurement schedules make sense, given factors such as shelf life of products and storage capacity relative to patterns of use. The indicator shows only whether the plan is consistent with historical use and anticipated future consumption patterns at a particular point in time. It does not measure whether the plan is actually carried out. To determine whether products are in fact procured and delivered as planned, one must monitor over time to see whether shipments of goods actually arrived at the expected times and in the right quantities. Program staff members can manually monitor small numbers of products, but they may require an automated tracking system (e.g., PipeLine, available from www.deliver.jsi.com) in programs managing many products. 22 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE The indicator may measure a national program, individual programs, or levels where procurement plans are prepared. RELATED INDICATORs • Existence of a procurement plan for each product (whether or not it maintains adequate stock) • Percentage of shipments that arrive on time • Percentage of shipments that arrive in the right quantities. Table 1. Instrument for Measuring Adequacy of Procurement Planning Established maximum stock level (if applicable) = _______ months Month (1) Beginning Balance (2) Quantity Received (3) Supplier (4) Estimated Consumption (5) Stock Adjustments (6) Ending Balance (7) Stock in Months (8) 01-Jan 01-Feb 01-Mar 01-Apr 01-May 01-Jun 01-Jul 01-Aug 01-Sep 01-Oct 01-Nov 01-Dec 02-Jan 02-Feb 02-Mar 02-Apr 02-May 02-Jun 02-Jul 02-Aug 02-Sep 02-Oct 02-Nov 02-Dec MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 23 GuIDELINEs fOR COMPLETION • In column 6, “Stock Adjustments,” include the projected amount of lost, expired, and damaged prod­ uct for each month. • Column 7, “Ending Balance,” is calculated by adding columns 2 and 3, subtracting column 5, and adding or subtracting column 6, depending on the direction of the adjustment. • Column 8, “Stock in Months,” is calculated by dividing the ending stock balance (column 7) by the projected average monthly consumption (normally calculated as the average of the previous three, six, or 12 months of consumption from column 5). For the most accurate calculation, use PipeLine soft­ ware (available on the Internet at deliver.jsi.com) or a computerized spreadsheet.1 � Table � is adapted from John Snow, Inc.’s PipeLine software, which is available on the Internet at www.deliver.jsi.com. 2� MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATOR: Percentage of Purchase Orders PlacedThat Are filled Correctly by External supplier (supplier fill Rate) DEfINITION This indicator is defined as the percentage of purchase orders made to an external supplier that are filled correctly by the external supplier in term of items and quantity. number of orders completed correctly × 100total number of orders This indicator can be calculated for any supplier that processes purchase orders made by a requesting facil­ ity. It can be measured over any time period, but one year is usually used. An order is defined as one or more products formally ordered from a single source at a given time. An order is filled correctly when the items and amounts supplied are exactly the same as the items and amounts requested by a facility. DATA REquIREMENTs • Purchase orders made with the supplier showing items and amounts requested • Shipping invoices or other such documents that show items and amounts shipped (or physical count of received shipment) • Dates that the purchase orders were received by the supplier and dates when they were shipped. DATA sOuRCEs Review of records at facility or supplier. PuRPOsE AND IssuEs This indicator measures the ability of the supplier to fill purchase orders correctly in terms of items and quantity. Shipments should always be checked against the shipping notice and the purchase order. What was shipped may not necessarily be what was ordered. Suppliers should be expected to perform at a high level. Even though a supplier may supply products only a few times a year, in most cases the supplier should be expected to fill orders completely or almost completely. If even one order has an error, an investigation of the causes is warranted. Suppliers that are routinely noncompliant should be replaced. If replacing a supplier is not possible, it may be necessary to identify which items are causing the most problems and find another mechanism for obtaining those items (perhaps through a donor). RELATED INDICATOR • Percentage of purchase orders placed that are filled correctly by external supplier and are on time. This indicator measures the ability of the supplier to fill purchase orders correctly in terms of items and quantity and to ensure that they are delivered within a specified delivery period. Other criteria can be added to the indicator to specify that a delivery order is compliant. In some cases, as many as 10 different criteria are used to evaluate whether an order is compliant. Orders that are compliant are considered to be “perfect,” and what is being measured is the percentage of perfect purchase orders. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 25 INDICATOR: supplier Lead-TimeVariability DEfINITION This indicator is defined as the mean of the absolute percentage differences between the lead time fore­ casted for purchase orders and the actual lead time for those same purchase orders. ∑APDi n where APD = | forecasted lead timei – actual lead timei | × 100i actual lead timei This indicator can be calculated for any supplier that supplies products to the requesting facility. It can be measured over any time period, but one year is usually used. It is usually measured in days. DATA REquIREMENTs • Shipping invoice or other such document that shows date when the purchase order was received by the supplier • Customs declaration showing the date the order was received in the country. DATA sOuRCEs Review of records at the supplier or customs or at the requesting facility (if available). PuRPOsE AND IssuEs During quantification exercises, it is important to estimate the expected lead time to determine when the next procurement cycle should begin. If the forecasted lead time differs significantly from the actual lead time, stock excesses or shortages may occur. It is important that these two figures be as close as possible. This indicator measures only the supplier’s lead time, however; it does not measure the total purchase order cycle time. The total purchase order cycle time is defined as the time from when the quantification of the purchase order begins until the products are received by the warehouse. It includes time on the front end to put together the purchase order and time on the back end to get the item from the port to the warehouse. 26 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATOR: Percentage of Average International Price Paid DEfINITION This indicator measures the unit cost per item charged by an external supplier as a percentage of the aver­ age international unit price. average unit cost of item × 100average international unit cost of item This indicator can be calculated for any supplier that supplies products to a requesting facility. It can be measured over any time period, but one year is usually used. DATA REquIREMENTs • Invoices from the supplier showing unit prices of items purchased • Average international unit costs for items purchased • Dates that the orders were received by the supplier and dates when they were supplied (related indicator). DATA sOuRCEs Review of invoices at the supplier or at the requesting facility (if available). For international unit costs, the Management Sciences for Health (MSH) International Drug Price Indicator Guide and International Dispensary Association (IDA) catalogs can be referenced. PuRPOsE AND IssuEs This indicator measures the cost of items procured relative to the average international price paid. The lower the percentage of average international price paid, the more the cost savings. Conversely, if the indicator is greater than 100 percent, the country is paying a premium on the average international prices. The MSH International Drug Price Indicator Guide can be referenced for the most current average inter­ national prices for pharmaceuticals. This indicator can be used to measure the costs of items within a procurement or across many procurements. If more than one procurement is being analyzed, the average unit costs of each item across the procurements should be used. RELATED INDICATOR • Percentage of international total price paid (including freight and insurance). If freight and insurance make up a significant portion of the total cost of previous procurements, those costs need to be applied to the average unit cost of the item procured and the average international unit cost of the same item. The new indicator then becomes the average total delivered cost of the item procured over the average international total price paid for the same item. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 27 28 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATORS FOR MEASURING WAREhOUSING AND INVENTORY MANAGEMENT PERFORMANCE INDICATOR: Order fill Rate (or Percentage of Orders PlacedThat Are filled Correctly) DEfINITION This indicator is defined as the percentage of all customer orders placed to a distribution source over a period of time that are filled correctly in terms of items and quantities of those items. number of orders completed correctly × 100total number of orders This indicator can be calculated for any facility that processes requests and supplies commodities to lower- level facilities. It can be measured over any time period, but one year is usually used. An order is defined as one or more products formally requested from a single source at a given time. An order is filled correctly when the items and amounts supplied are exactly the same as the items and amounts requested. DATA REquIREMENTs • Requisition vouchers or similar order forms submitted by the requesting facility to the supplying facil­ ity, showing items and amounts requested • Same requisition forms or similar forms issued by the supplying facility, showing items and amounts supplied (alongside amounts originally requested) • Dates when the orders were received and dates when they were supplied. DATA sOuRCEs Review of records at facility and observation of items and amounts shipped. PuRPOsE AND IssuEs This indicator measures the ability of the supplying facility to fill requests for resupply correctly in terms of items and quantity. It differs from the indicator for percentage of facilities that receive the quantity of products ordered in that it evaluates a facility’s ability to fill an order in its entirety (as opposed to its ability to fill an item in that order). As such, it is not useful in identifying which items are most often in error (possibly because of rationing). MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 29 Two methods are generally used for collecting data for this indicator. The supplying store can review each request just before it is shipped out to a customer (by actually counting the amounts packed to ship). The other generally used method is to have a person at the requesting facility check the items and amounts shipped (also by counting) and report back to the supplying facility. The error rates generated using this indicator should be evaluated with care. For distributing facilities that received 100 or more orders over a one-year period, error rates of two percent or higher are considered unacceptable. However, for distributing facilities that process fewer than 100 orders, the percentage error rate is not as important as the absolute number of orders that were filled incorrectly (e.g., in cases where only 20 orders are filled each year, one incorrect order will result in a five percent error rate, which in this case may or may not be serious). If error rates for this indicator are observed to be high, an investigation of the causes is warranted. The following are possible causes of high error rates: • Originally requested items and quantities (to pick) were entered incorrectly into an automated system. • Incorrect items or amounts were picked by the picker. • Insufficient amounts were in stock to fill the order, and therefore the item was rationed (either by the data entry person or the picker). • Amounts requested were felt by the supplying store to be incorrect and were not filled as requested. The latter two causes should be indicated on the request or issue voucher in a notes column next to the respective commodity being issued. If this entry is made mandatory, it will help focus attention on prob­ lems with data entry or picking. If it is not made mandatory, evaluation of each product may be necessary, using the indicator for percentage of facilities that receive the quantity of products ordered to deduce whether rationing is occurring. RELATED INDICATOR • Percentage of orders placed that are filled correctly and are on time. This indicator is defined as the percentage of all orders placed to a distribution source over a period of time that are filled correctly in terms of items and quantities of items and are delivered to the requesting facility within a predetermined amount of time. • Order fill rate by product. For many logistics systems the order fill rate for entire orders will often approach zero. There are many possible reasons which include rationing, stockouts at the central or regional levels, and delayed or cancelled shipments. For this reason evaluators may choose to look at order fill rates by product or by program (e.g., RH, HIV, TB, STI). This will provide program managers with a clear picture of which program and or product is problematic. 30 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATOR: Inventory Accuracy Rate (or Accuracy of stock Balance for Inventory Management) DEfINITION This indicator measures the accuracy of stock balances recorded in a stock ledger, bin card, or automated system (i.e., ledger amount equals physical count) over a range of items as a percentage of stock balances reviewed for accuracy. number of items where stock record count equals physical stock count × 100total number of items counted This indicator can be calculated for any facility that manages the items in question. If physical invento­ ries are done once yearly, then this indicator is calculated whenever a physical inventory is taken. If cycle counting is used by the facility, then this indicator can be measured over one or a number of cycle counts (e.g., over all cycle counts done in one month). DATA REquIREMENTs • Physical counts of items in the facility • Automated system, stock ledger, bin card, or other inventory management recording instrument on which stock balances are maintained. DATA sOuRCEs Facility survey or site visits to facilities being monitored and facility inventory management records. PuRPOsE AND IssuEs This indicator measures the accuracy of data on product stock levels at a facility and provides information on how accurately the facilities are tracking their inventories. Having accurate stock-on-hand values is essential for forecasting and procurement exercises as well as for proper picking and distribution. This indicator differs from the indicator for accuracy of logistics data for inventory management in that this indicator measures a facility’s ability to maintain accurate stock records for all products (rather than its ability to maintain accuracy for a single product). As such, it is not useful in identifying which prod­ ucts are in error. This indicator is generally calculated during a physical inventory. Physical inventories can be done on a fixed schedule (e.g., all items are counted annually), or they can be done with higher frequency such that each item is counted according to its own schedule (e.g., aspirin is counted quarterly; norplant is counted annually). Annual physical inventories are likely to reveal more items in error than are counts done with higher frequency. The following are possible reasons for poor record accuracy (over 20 percent if annual, over 5 percent if cyclical): • Incorrect recording of amounts received and issued (by picker if manual system, by data entry person if automated system) • Incorrect items or amounts picked by the picker MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 31 • Incorrect counting of amounts received • Arithmetic errors (by data entry person). RELATED INDICATOR • Accuracy (within tolerated limits) of stock balance for inventory management. Because small deviations in accuracy over a long period of time are not significant, a tolerated accuracy level may be used. This indicator is defined as the percentage of items counted that are outside the book balance, plus or minus a tolerated amount (usually about a five-day supply). This indicator is useful only if the consumption rates can be easily calculated and, therefore, can be used in the indicator. • Percentage discrepancy between stock record balance and physical inventory (by product). Evaluators may find this indicator useful when looking at different programs. For instance, there may be poor record keeping practices for contraceptives but good practices for ARVs. 32 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATOR: Percentage of stock Wasted due to Expiration or Damage DEfINITION This indicator is defined as the percentage of counted stock for an item that is unusable because of expira­ tion or damage. unusable physical stock count × 100unusable plus physical stock count This indicator can be calculated for any facility that manages the commodities of interest. It can be measured over any time period but is usually calculated whenever a physical inventory is taken. DATA REquIREMENTs • Physical counts of usable and unusable items • Automated system, stock ledger, bin card, or other recording instrument on which stock balances are maintained. DATA sOuRCEs Facility survey site visits to facilities being studied. PuRPOsE AND IssuEs For the purposes of this indicator, unusable (i.e., wasted) items are defined as those that have expired or have been damaged in such a manner that they are no longer safe to use. This indicator measures the ability of the warehouse to practice FEFO methods and properly manage commodities (i.e., without incurring damage). Reducing wastage rates not only saves the organization money but also helps ensure customers receive quality products. High expiration rates can result from several factors that depend on the commodity. Some commodi­ ties (like test kits) may expire because they have very short shelf lives. Other commodities may have been procured in amounts too large to be properly distributed before expiration. But if neither of those factors is relevant, the cause is most likely improper inventory control. Items with high wastage rates attributable to expiration (above 5 percent for an annual physical count, above 1 percent for each cyclical physical count) should be checked to see if shelf life is abnormally low. If so, these items should be monitored very closely and probably moved to the front of the shelf or real­ located to facilities with higher consumption rates for those products. If shelf life is not an issue, check to see how many months of supply are in stock. If the months-of-stock for a commodity is greater than the months left before stock expires, then improper procurement or ordering is an issue and stock will have to be reallocated to other facilities. This indicator provides an important but incomplete picture of wastage. The full effect of wastage on the program may be greater than that suggested by the indicator. Facilities with high wastage rates that dispose of products with or without proper records may look better than facilities with low wastage rates but with expired or damaged products still in the storeroom. Wastage can occur through loss as well as expiration or damage. This indicator does not incorporate wastage caused by loss. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 33 RELATED INDICATORs • Percentage of unusable stock that is unusable because of expiration. This indicator measures the portion of unusable stock that is unusable because of expiration only. It is given as follows: unusable physical stock count attributable to expiration × 100total unusable physical stock count • Percentage of unusable stock that is unusable because of damage. This indicator measures the portion of unusable stock that is unusable because of damage only. It is given as follows: unusable physical stock count attributable to damage × 100total unusable physical stock count • Value of unusable stock as a percentage of total item purchases. This indicator measures the portion of stock found to be unusable over a given period of time as a percentage of the total drug purchases made during that same period of time. It is given as follows: value of unusable physical stock × 100value of commodity purchases 3� MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATOR: OrderTurnaroundTime DEfINITION This indicator is defined as the average amount of time it takes for a facility to fill an order from the date the order is received by the facility until the date the order is shipped to the customer. n ∑OTTii=1 OTT = number of days to process thei order or order turnaround time This indicator is usually recorded in days. It can be calculated over any period of time but usually is recorded over one year. DATA REquIREMENTs Requisition and issue vouchers showing the date the order was received by the facility and the date the order was shipped out. DATA sOuRCEs Requisition and issue vouchers of the supplying facility. PuRPOsE AND IssuEs This indicator measures the efficiency with which requests are processed because it measures only the time between when the request was received by the distribution source and the time the order was actually shipped. Improving turnaround time will improve customer satisfaction and, more important, improve productivity and thus reduce costs for staff time. In addition to measuring the mean turnaround time, it is possible to break this turnaround time into its component parts (pick time, loading time) and deter­ mine where most of the processing time is spent and, thus, where resources could be reallocated. Each of these component parts could be an indicator by itself. In cases in which the facility has established a turnaround time, this indicator can be adjusted to measure the percentage of orders shipped on time. It may also be useful to group orders by size—large, medium, and small. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 35 INDICATOR: InventoryTurnover Rate DEfINITION This indicator is defined as the total value of items distributed divided by the average value of the inven­ tory managed over a given time period (usually one year). total value of items distributed average value of inventory DATA REquIREMENTs Records of the price paid for inventory. DATA sOuRCEs Shipping invoice or requisition and/or issues voucher. PuRPOsE AND IssuEs This indicator measures the number of times the inventory turns over (or is replaced) in a given time period. It is measured in number of turns. At facilities that have been functioning for long periods and have streamlined their procurement processes, the values calculated for this indicator should range anywhere from 6 to 12. In general, the higher the ratio, the lower the average inventory level (and average holding cost). Average unit prices and average inventory levels for the period in question should be used in the calculation. At warehouses with high maximum levels, the ratios can be expected to be low for these products, espe­ cially at central-level warehouses functioning in countries with unpredictable procurement. 36 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATORS FOR MEASURING DISTRIbUTION PERFORMANCE INDICATOR: Ratio of Distribution Cost toValue of Commodities Distributed DEfINITION This indicator is defined as the ratio of the cost of distributing or transporting orders to the value of those orders. cost of transporting commodities value of commodities transported This indicator is usually calculated for each delivery route that is supplied. It can be measured over any time period, but one year is usually used. DATA REquIREMENTs • Information on costs associated with each delivery of an order or orders on a delivery route (includ­ ing, but not necessarily limited to vehicle fuel, maintenance and depreciation costs; and person-hours devoted to delivery) • Requisition forms or similar forms issued by the supplying facility showing items and amounts supplied • Information on the unit costs of items supplied on each requisition form. DATA sOuRCEs Review of vehicle logs, staff records, and requisition vouchers at the supplying facility or observation of items and amounts shipped. Unit costs can be averaged from purchase orders during the time period. PuRPOsE AND IssuEs This indicator measures costs associated with delivering commodities to each subordinate facility or facili­ ties on a delivery route or costs associated with distribution as a whole. The calculated ratios indicate the relative cost of supplying commodities to each facility or facilities. When delivery routes are being evaluated, high ratios of cost of delivery compared to value of the delivery may suggest that the review periods for these facilities be lengthened or else that shipments to these facili­ ties be consolidated with shipments to other facilities. All other factors being equal, the percentage cost of routine delivery to value of commodities should not be very large. The percentage can be expected to be much greater for emergency shipments. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 37 This indicator is meant to be used in situations in which transportation costs can be easily ascertained. Central and regional warehouses would be suitable candidates. However, lower-level facilities that have to improvise delivery to subordinate facilities most probably cannot be evaluated using this indicator. For those routes that can be evaluated, unit costs for commodities delivered can be based on the average unit cost during the period being monitored. Maintenance costs for vehicles must be allocated to each facility proportionate to the percentage of overall miles or kilometers traveled that were dedicated to deliv­ ery to the facility. Depreciation costs can be determined from standard depreciation tables for the make and model of vehicle being used to deliver the commodities. Depreciation costs must also be apportioned accordingly. If warehouses are being studied and they use vehicles to make deliveries, then data on fuel and mainte­ nance costs should be available through vehicle logs and records. If depreciation costs cannot be deter­ mined, then they can be excluded. RELATED INDICATOR • Ratio of distribution cost to value of commodities distributed (across all facilities supplied). This indicator is defined as the ratio of the cost to a warehouse of distributing or transporting all orders to all its subordinate facilities compared to the value of those orders. cost of transporting commodities value of commodities transported This indicator is usually calculated for each warehouse that is responsible for making deliveries. It can be measured over any time period, but one year is usually used. The indicators for different warehouses can be compared to evaluate performance. 38 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE INDICATOR: Average DeliveryTime DEfINITION This indicator is defined as the average time it takes to deliver an order to a facility. n ∑DTii=1,n DTi, n = delivery time This indicator is usually calculated for each facility that is supplied by an upper-level warehouse. It can be measured over any time period, but one year is usually used. It can be compared with delivery times recorded using alternative routing. DATA REquIREMENTs • Vehicle logs showing time out of warehouse and time into a facility for each delivery • Shipping notices that show time out of warehouse and time into a facility for each delivery or requisi­ tion and issue vouchers (RIV). DATA sOuRCEs Review of vehicle logs and shipping notices or RIVs. PuRPOsE AND IssuEs This indicator measures the average amount of time it takes to deliver an order from a supplying store to a subordinate store. If alternative routes are available to deliver commodities to the same location, these routes may be tried, and the average delivery time may be calculated for each. Average delivery times for each route can be compared to see which is most efficient. It should be noted that more efficient routes may not necessarily be more cost-effective. In some cases, a route with an average delivery time less than that of other available routes may actually cost substantially more because of tolls and other transport costs. RELATED INDICATOR • Variance between delivery schedule and actual delivery. • Average delivery time per warehouse. This indicator is defined as the average delivery time measured across all deliveries made by a warehouse to all facilities it supplies. It is calculated for each warehouse responsible for making deliveries. Average deliv­ ery times for each warehouse can be compared to evaluate performance. Care must be taken to include all factors that may influence delivery time before concluding that one warehouse is functioning better than others. It may also be useful to separate emergency orders from routine orders and to determine the aver­ age delivery time for each kind of order. MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE 39 �0 MONITORING AND EVALUATION INDICATORS FOR ASSESSING LOGISTICS SYSTEMS PERFORMANCE For more information, please visit www.deliver.jsi.com DELIVER John snow, Inc. 1616 N. Fort Myer Drive, 11th Floor Arlington,Va 22209 USA Phone: 703-528-7�7� Fax: 703-528-7�80 www.deliver.jsi.com

View the publication