Guide to Conducting Supply Chain Assessments Using the LSAT and LIAT

Publication date: 2011

Guide to Conducting Supply Chain Assessments Using the LSAT and LIAT APRIL 2011 This publication was produced for review by the U.S. Agency for International Development. It was prepared by the USAID | DELIVER PROJECT, Task Order 1. Guide to Conducting Supply Chain Assessments Using the LSAT and LIAT The authors' views expressed in this publication do not necessarily reflect the views of the U.S. Agency for International Development or the United States Government. USAID | DELIVER PROJECT, Task Order 1 The USAID | DELIVER PROJECT, Task Order 1, is funded by the U.S. Agency for International Development under contract no. GPO-I-01-06-00007-00, beginning September 29, 2006. Task Order 1 is implemented by John Snow, Inc., in collaboration with PATH; Crown Agents Consultancy, Inc.; Abt Associates; Fuel Logistics Group (Pty) Ltd.; UPS Supply Chain Solutions; The Manoff Group; and 3i Infotech. The project improves essential health commodity supply chains by strengthening logistics management information systems, streamlining distribution systems, identifying financial resources for procurement and supply chain operations, and enhancing forecasting and procurement planning. The project also encourages policymakers and donors to support logistics as a critical factor in the overall success of their health care mandates. Recommended Citation USAID | DELIVER PROJECT, Task Order 1. 2011. Guide to Conducting Supply Chain Assessments Using the LSAT and LIAT. Arlington, Va.: USAID | DELIVER PROJECT, Task Order 1. Abstract Supply chain assessments are conducted periodically to support the development of logistics systems. As assessments become a standard part of overall logistics management, the capacity of in-country stakeholders to drive the assessment process needs to increase. As a guide to conducting supply chain assessments with the Logistics System Assessment Tool (LSAT) and the Logistics Indicators Assessment Tool (LIAT), this document is meant to support capacity building and supplement the lessons that evaluators have learned from their own experience. Cover photo: District Supervisor, Mr. Mosses, reviews logistics data with a staff person during a routine monitoring and data collection activity at a dispensary in Tabora, Tanzania in 2009. USAID | DELIVER PROJECT John Snow, Inc. 1616 Fort Myer Drive, 11th Floor Arlington, VA 22209 USA Phone: 703-528-7474 Fax: 703-528-7480 Email: askdeliver@jsi.com Internet: deliver.jsi.com Contents Acronyms. vii Acknowledgments . ix Overview .1 Background.1 Purpose .1 Intended Audience.1 Sections in this Guide .2 Planning an Assessment. 3 Before Conducting a Survey .3 Using the LSAT. 9 Planning .9 Data Collection .12 Data Analysis.14 Monitoring .14 Using the LIAT .17 Survey Design .17 Sampling .19 Survey Preparation .29 Data Collection Training.31 Data Collection .36 Data Entry .39 Data Analysis.40 Presenting Results.43 Assembling Key Information .43 Presenting the Results .43 Storing Data and Results for Follow-On Activities .44 v Figure 1. Costs Associated with Surveys Using the LSAT and LIAT .5 Tables 1. Strengths and Weaknesses of Commonly Used Facility Survey Designs .18 2. Commonly Used Probability Sampling Types.26 3. Example of the Facility Unique Identifier Number Scheme .28 4. List of Sample Indicators for Evaluating a Logistics System .41 Boxes 1. Chapter Summary: Planning an Assessment . 7 2. Suggested Discussion Session Timing.12 3. Suggested Pairings and Guidelines for Group Work.13 4. Chapter Summary: Using the LSAT.15 5. Examples of Acquiring Sample Sizes of Statistical Significance .23 6. Chapter Summary: Using the LIAT.42 7. Chapter Summary: Presenting Results.44 vi Acronyms AIS HIV/AIDS Indicator Survey DEFF design effect DHS Demographic and Health Survey FPC finite population correction GIS geographic information system GPS global positioning system HQ headquarters ICC intraclass coefficient LIAT Logistics Indicator Assessment Tool LMIS logistics management information system LSAT Logistics System Assessment Tool M&E monitoring and evaluation MICS Multiple Indicator Cluster Survey MIS Malaria Indicators Survey MOH Ministry of Health MOSH months of stock on hand NGO nongovernmental organization PDA personal digital assistant RHS International Reproductive Health Survey SOP standard operating procedure SOW scope of work SPA Service Provision Assessment SPARHCS Strategic Pathway to Reproductive Health Commodity Security vii viii Acknowledgments Completion of the Guide to Conducting Supply Chain Assessments Using the LSAT and LIAT was made possible through the technical and editorial contributions of Ethan Collins, Erika Ronnow, Jaya Chimnani, Suzy Sacher, Dana Aronovich, Andrew Inglis, Ariella Bock, Emma Stewart, Julia Byington, and Paul Crystal. In addition, Trisha Long and Mike Tracy were instrumental in developing and editing the Access database that is included in the toolkit this document accompanies. It cannot be emphasized enough that all of these individuals are indebted to the many program stakeholders and health facility personnel throughout the developing world who have helped guarantee the success of logistics assessments conducted with support from the DELIVER project and the USAID | DELIVER PROJECT, and have added to the experiences that form the basis of this guide. ix x Overview Background Since being introduced in 2001 by the DELIVER project, the Logistics System Assessment Tool (LSAT) and Logistics Indicators Assessment Tool (LIAT) have been standard methods for conducting evaluations of health commodity supply chains in the developing world. Most often used jointly, the LSAT is designed to facilitate a comprehensive quality assessment of the separate components that make up a logistics system, and the LIAT is used to assess how well the system is functioning; that is, whether the right quantities of the right products are available to all customers at the right time, in the right place, and in the right condition. Both tools were developed for use by ministries of health, NGOs, and other supply chain stakeholders. This guide was written to provide details on the beginning-to-end process of conducting assessments whose results can effectively inform decision making and ultimately contribute to improved health commodity security. Purpose Supply chain assessments are conducted periodically to support routine management and strengthening of logistics systems. As assessments become a standard part of overall logistics management, the capacity of in-country stakeholders to lead the assessment process needs to increase. As a guide to conducting supply chain assessments with the LSAT and LIAT, this document is meant to support capacity building and supplement the lessons that evaluators have learned from their own experiences. It serves as the introduction to a toolkit that includes—  Standard LSAT and LIAT documents.  Monitoring and Evaluation Indicators for Assessing Logistics Systems Performance.  Data Entry Tool and Database (Access), Data Entry Guidelines, and Double Data Entry Quality Control Tool (Excel).  Guidelines for Collecting and Using GPS Data for Monitoring and Evaluation Exercises and download for GPS Trackmaker software. The chapters herein focus on determining which type of assessment can best meet specific information needs and how to properly estimate the resources required for a successful assessment. They also emphasize that a better understanding of the assessment process and proper planning will save time and money and produce more useful results. Intended Audience This guide is a companion to the LSAT and LIAT and is intended for people who manage or conduct assessments using either tool. These may include program managers, technical advisors, personnel from various nongovernmental organizations (NGOs), and ministry of health (MOH) officials. 1 This guide also provides important background information to decision makers on how supply chain assessments are conducted to meet specific information needs and produce valid results. But the information presented is not necessarily targeted at decision makers, particularly because the process of using assessment results to develop recommendations and action plans is not discussed in depth. Sections in this Guide This guide is divided into four sections— 1. Planning an assessment 2. Using the LSAT 3. Using the LIAT 4. Presenting results A variety of supplemental information, model materials, and other tools are provided in the appendices of this guide, as well as in the larger toolkit mentioned above. 2 Planning an Assessment Supply chain assessments are carried out for a number of reasons, including—  Determining, as part of a pilot test, whether the commodity needs of a new program can be met reliably.  Identifying logistics problems related to a particular product category, such as family planning commodities or products used to prevent, diagnose, and treat malaria.  Tracking logistics performance over the course of a program.  Understanding how program changes—such as implementation of a campaign to increase service demand—affect the supply chain.  Improving supply chain performance and, ultimately, customer service. Before Conducting a Survey Determining whether an assessment is appropriate and feasible depends on when the assessment is being conducted; the availability of adequate financial, human, and material resources; whether similar information can be obtained from already-existing sources; and stakeholder agreement. When should an assessment be conducted? Supply chain assessments are as important for measuring changes in logistics system performance as they are for program planning. They should ideally be conducted at the beginning, mid-term, and end of a program cycle as a way to track the success of interventions and identify issues. Many programs choose to conduct only end-of-program cycle surveys. Yet, without robust data captured at the beginning of an intervention, the amount of useful information from a stand-alone, end-line survey will be more limited, and the results will be more open to misinterpretation. Mid-term assessments can be conducted once or at select intervals during the life of a program cycle to help determine which program interventions are successful and which may need to be changed or better targeted. Whenever possible, results from mid-term assessments should be available to inform each phase of work planning and budgeting. The timing of assessments should also take into consideration other major logistics work that could be informed by survey results. This includes forecasting and system design workshops at the country level and the submission of key status reports by technical assistance providers. Are the LSAT and LIAT always used jointly? As suggested in the Overview, the combined effect of conducting a survey with the LSAT and LIAT is an understanding of how well a logistics system is structured and how efficient it is in facilitating the availability of commodities. Although both tools are ideally used to form a complete picture of the supply chain, a number of factors may influence the use of only one. 3 A well-functioning logistics management information system (LMIS) will often produce much of the same data on commodity availability and products flows that could be gained from an assessment in which the LIAT is used. Stakeholders with only limited funds for conducting evaluations may choose to analyze LMIS data instead of conducting a facility-based survey as a cost-savings measure. It is important to note, though, that such a decision assumes the LMIS contains accurate information from all facilities in a system. For its part, the LIAT is not only used to collect data on product availability, but also to support these findings. The LIAT helps determine:  when stockouts have occurred, as well as how long they lasted;  how well LMIS reporting requirements are being met;  what logistics training and supervision facility staff have received;  whether storage conditions are sufficient to ensure the quality of commodities; and  whether LMIS records are being maintained routinely and accurately. When only the LSAT is used for an evaluation, it is usually advisable for assessors to conduct a limited number of site visits to accumulate anecdotal evidence that supports the conclusions they reach using the LSAT. Similarly, LIAT-only assessments can be done when there is confidence that the logistics system is functioning and well organized, or simply because stakeholders are primarily interested in knowing about the use of logistics management systems and product availability. What resources are needed? The scope of an assessment depends largely on the resources that are available. All relevant stakeholders should develop and approve accurate budget estimates beforehand to better ensure that intended objectives are met. Additionally, it is necessary to recruit personnel who will commit to participating throughout the length of the survey. It is therefore important to understand the amount of time that will be needed to organize an assessment, train data collectors, collect data, enter data into a database, analyze data, debrief stakeholders, and write a final technical report. A detailed activity schedule for a five-week assessment using both the LSAT and LIAT is provided in Appendix 1. The activity schedule can be summarized as follows: Week 1  Implementation of the system assessment (LSAT)  Analysis of results from the system assessment (LSAT) Week 2  LIAT training (LIAT) Weeks 3 and 4  Data collection and data entry for the facility-level assessment (LIAT) Week 5  Analysis of results from the facility-level assessment (LIAT)  Presentation of preliminary results from the system assessment and facility-level assessment (LSAT and LIAT) 4                                                                                                                                                                                                     As shown in Figure 1, all assessments entail a certain number of required costs and can also include a number of supplementary elements. When the LIAT and LSAT are used concurrently, some cost items may be paired to produce cost savings, such as international travel costs for external technical assistance. It is also important to remember that unforeseen costs almost always arise in assessments of any type, so a contingency fund should be planned in addition to the base budget. Because the scope and coverage of supply chain assessments can vary widely, it is difficult to define a typical cost for conducting surveys with the LSAT and LIAT. Recent facility-level assessments (LIAT) supported by the USAID | DELIVER PROJECT have generally required total costs of around US$100,000. This estimation reflects data collection in 80 to 100 facilities and includes external technical assistance. Costs would be lower for programs employing only local human resources. The latest system assessments (LSAT) conducted by the project have cost approximately U.S. $25,000. Figure 1. Costs Associated with Surveys Using the LSAT and LIAT REQUIRED SUPPLEMENTARY Facility‐level Assessment (LIAT) •Activity materials. •Meeting venue, including lunch and coffee breaks. •Per diem and transportation for participants. •Travel, per diem, and salary for a short‐term technical assistance provider. •Advertising for survey coordinator, data collectors, and data entry personnel. •Contracts with survey coordinator and data entry personnel, as appropriate. •Travel, per diem, and salary for a short‐term technical assistance provider. •GPS devices, GPS software, and GIS software (usually supplied by the technical assistance provider). •Activity materials. •Training venue, including lunch and coffee breaks. •Accommodation, per diem, and transportation for training participants. •Transportation for data collection, monitoring, or supervision field visits. •Accommodation and per diem for data collectors and monitors. System Assessment (LSAT) What information already exists? One of the more important initial steps in determining the need and feasibility of a supply chain assessment is undertaking secondary research to determine what relevant data and information already exist. For example, sufficient data for a situational analysis (but not for a more complex longitudinal study) can often be found in surveys already conducted by other organizations or, as noted above, through an LMIS or supervisory system. Some helpful sources of country-level evaluations are the—  Demographic and Health Survey (DHS): www.measuredhs.com 5  International Reproductive Health Surveys (RHS): http://www.cdc.gov/reproductivehealth/surveys/  Service Provision Assessment (SPA) Survey: www.measuredhs.com/aboutsurveys/spa  HIV/AIDS Indicator Survey (AIS): www.measuredhs.com/hivdata  Malaria Indicators Survey (MIS): www.measuredhs.com/aboutsurveys/mis  Multiple Indicator Cluster Survey (MICS): http://www.childinfo.org/mics  The Strategic Pathway to Reproductive Health Commodity Security (SPARHCS): http://deliver.jsi.com/dhome The above-listed surveys are conducted in many countries on a recurring basis and provide useful data for making comparisons. Surveys of this kind also have database and GPS coordinates of health facilities available for download. These data may be used to compare results collected in the LIAT and, through the matching of GPS coordinates, even compare data for individual sites. Secondary research can also help understand background issues that affect a logistics system, such as product demand and the rate of commodity usage, as well as potential future sources of donations. Familiarity with the country’s health policies, annual budget, standard treatment guidelines, and list of essential medicines will help to clarify constraints on the supply chain system under evaluation. A country’s strategic plan (as it relates to supply chain management) will also help evaluators understand the logistics system before the assessment begins and will help focus the survey on capturing the most important information needs. A better understanding of the factors affecting the supply chain may lead the evaluation coordinator or country program to adapt the LSAT or LIAT questionnaire to extract more useful information. Adaptations most often involve adjusting questionnaires to refer specifically to product lists and categories, the precise logistics forms that are used, correct administrative zone divisions, and correct titles for program stakeholders and role players. What are stakeholders’ roles? Ownership of the assessment process and its results by ministries of health and other program stakeholders is crucial to success, even when a survey is being conducted on behalf of a donor or another outside group. Involving stakeholders from the beginning of assessment planning is the best way to foster client ownership and manage expectations. For example, if a client wishes to conduct an exhaustive assessment at a low cost, it is important to explain what information a survey can capture, particularly in relation to the amount of available financing for the activity. It is especially important to remind stakeholders to collect only data that will be used for decision making to avoid the expense of accumulating information that will ultimately not be useful. Helping a client set realistic expectations through regular involvement in the assessment planning process is a critical step in allowing a survey to start on time and stay on budget. During the planning stage, it should be determined which organizations will provide assistance, including vehicles, personnel, and financing. Regardless of material or financial input, primary stakeholders should provide personnel, if at all possible. Including stakeholders in the assessment will allow their staff to gain a perspective of the situation on the ground, and will foster a sense of ownership over the information and responsibility to act on the results. It is suggested that one organization be the main implementer of the assessment to avoid confusion over roles and responsibilities. In most cases, the MOH is the agency that is requesting the assessment; therefore, it plays a key role in its planning and oversight. However, well-coordinated 6 implementation is vital to the success of an assessment, and the entity that has this responsibility should know exactly what is needed to ensure the appropriate amount of coordination and communication with the government ministry. Box 1. Chapter Summary: Planning an Assessment  Supply chain assessments should ideally be conducted at the beginning, mid-term, and end of a program cycle as a way to track the success of interventions and identify issues. The feasibility of conducting assessments at each of these points is influenced by the availability of adequate financial, human, and material resources, and stakeholder agreement.  Undertaking secondary research to determine what relevant information already exists can help reduce the time and cost of data collection in the field, and can provide some evidence for decision making when conducting an assessment is not practical.  All relevant stakeholders should develop and approve accurate budget estimates beforehand to better ensure that intended objectives are met. The budget will depend on the size and scope of the assessment.  Ownership of the assessment process and its results by ministries of health and other program stakeholders is crucial to success, even when a survey is being conducted on behalf of a donor or another outside group.  Among stakeholders, it is suggested that one organization be the main implementer of the assessment to avoid confusion over roles and responsibilities. 7 8 Using the LSAT The Logistics System Assessment Tool (LSAT) is used to conduct discussion groups and key informant interviews for assessing 11 elements that comprise public and private sector logistics system, and their enabling environments, including: 1. Organization and staffing 2. Logistics management information systems (LMIS) 3. Product selection 4. Forecasting 5. Procurement 6. Inventory control procedures 7. Warehousing and storage 8. Transport and distribution 9. Organizational support 10. Product use 11. Finance, donor coordination, and commodity security planning These elements are relevant to all logistics systems, whether in the public sector or private sector. As a diagnostic and monitoring tool, the LSAT can be used to complete annual assessments and contribute to work planning. Information collected using the LSAT is analyzed to identify issues and opportunities and, from those, outline further assessments if additional information is needed, as well as appropriate interventions. Because assessments using the LSAT are conducted and analyzed in successive years, the results can be used to monitor and improve system performance, and to provide critical data that can identify a country’s commodity security strengths and weaknesses. The LSAT can—  provide stakeholders with a comprehensive view of all aspects of a logistics system;  be used as a diagnostic tool to identify logistics and commodity security issues and opportunities;  raise collective awareness and ownership of system performance and goals for improvement;  be used by country personnel as a monitoring tool (to learn and continually improve performance); and  provide input for work planning, particularly when conducted within the three-month period prior to work planning exercises. Planning To produce effective results that can be used for decision making, it is essential for an assessment to fit the context of the supply chain it is evaluating, in terms of the specific information being sought with the LSAT, the method of collecting data, and the people who will provide those data. 9 Preparatory Research A thorough review of the LSAT questionnaire and research on certain assessment aspects should be conducted before discussions or interviews begin. Examples of documents that will be useful in preparatory research are standard operating procedures (SOPs), guidelines, policies that include logistics, and national drug lists. This information should be presented and validated during the course of the assessment. Choosing a Data Collection Method Choosing whether to use the LSAT to facilitate discussion groups, key informant interviews, or both will influence planning. Program managers and country counterparts should be consulted to determine which approach will be used. Large discussion groups may require sessions that last one to one-and-a half days to capture the breadth and depth of data required and to provide an adequate opportunity for full participation by all persons involved. If work planning is part of the exercise, it will further extend the time needed for the sessions. Using the LSAT as a guide for key informant interviews can take up to two weeks or more because of the time required to schedule and conduct multiple interviews with the people who have knowledge about the many components of the logistics system. Option 1: Discussion Groups Using the LSAT to facilitate discussion groups can be organized in different ways—  Joint discussion groups (recommended)  Both central-level and lower-level participants can be brought together in one session. This session will probably include 15–25 participants and will require skilled facilitation. This will probably take one to two days to complete, depending on the number of participants and the level of work planning included in the exercise.  Separate central-level and lower-level discussion groups  Central-level group sessions should include approximately 10–25 participants and are the minimum requirement for using discussion groups for information collection.  Lower-level group sessions generally comprise a cross-section of units (e.g., districts), although it may be necessary to select a different subset, such as a particular geographic area or units under a particular set of circumstances. This option will require at least one day at each site. Option 2: Key Informant Interviews Using the LSAT as an interview guide to collect information from numerous key informants can take one week or more, mainly due to the time it takes to schedule and conduct all interviews (usually at separate locations), ensure that the range of information sought is provided across the pool of interviewees, and consolidate data for a harmonious final report. One disadvantage to this approach is that it does not allow for group discussion between people working in different areas of the supply chain, which promotes an exchange of ideas and information, as well as consensus building around answers. If this approach is used, it is recommended that a stakeholders’ meeting be held to present and discuss the assessment findings. 10 Option 3: Discussion groups and key informant interviews A combination of data collection methods is often quite useful when using the LSAT. When group discussions do not involve all of the intended participants, or information gathered this way is not complete, key informant interviews can provide invaluable supplementary data. Key informant interviews can also be used a first step before group discussions, providing organizers with an opportunity to adapt the LSAT in advance in collaboration with experts on the local supply chain context. Selecting Participants To collect thorough and accurate data about each aspect of the logistics system, it is important to select the appropriate set of people to participate. For year-to-year use of the discussion group option, it is useful to work consistently with the same core group of participants as a way to build internal capability for conducting supply chain assessments and to improve the reliability of the data. It is best to consider already-existing groups, such as logistics committees, as a source of participants. Each participant should be knowledgeable about one or more of the 11 areas covered by the LSAT and, ideally, have hands-on experience with the logistics system at the level the participant is representing (i.e., central or lower level). Program managers should identify appropriate participants. It is advisable to consider international donors and Ministry of Finance personnel for the finance knowledge area. It is also useful to include someone with policy expertise because policy questions are incorporated into several sections. Central-level discussion groups or interviews should include participants with a knowledge base in all LSAT areas, but knowledge of organization and staffing, product selection, forecasting, and procurement (i.e., central-level functions) may be excluded at lower levels of the system if these functions are highly centralized. Field Visits It is recommended that facilitators or interviewers make a limited number of field visits. Field visits made prior to the discussion sessions and interviews will provide a sample of the current context or circumstances, which will add additional insight into the information collection. Visits made following the discussions/interviews offer an opportunity to further explore issues that were identified, enhance the quality of the information gathered, and allow for additional data collection. Individuals making the field visits can focus on unanswered LSAT questions; data that are mixed, unclear, or that have been disputed; and disparate or wide-ranging responses to questions. Program managers or country counterparts can help plan the appropriate number of field visits before and/or after the exercise. 11 Data Collection The methods for collecting data through discussion groups and key informant interviews are fairly similar, but activities for each are planned and organized differently. A flowchart for determining a data collection method for system assessments using the LSAT is available in Appendix B. Option 1: Discussion Groups As noted earlier, information-gathering sessions should be led by a skilled facilitator who is familiar with the LSAT. Field experience has also shown that having multiple recorders who are also familiar with the tool is more likely to result in capturing high-quality information. Facilitators should set the tone for the session by explaining how the participants’ input will be used and what they want to hear from each person about their area of knowledge, as well as how they see the technical areas relating to and impacting one another. In the sample agenda shown in Box 2, Organization and Staffing is covered through discussion with the entire group (plenary), followed by group work for the remaining sections. Box 2. Suggested Discussion Session Timing 8:30–9:00 AM Introduction, Objectives, and Agenda (plenary) 9:00–10:00 Organization and Staffing (plenary) 10:00–10:15 Break 10:15–1:00 Group Work by LSAT Component 1:00–2:00 Lunch 2:00–2:30 Group Work Presentation Preparation 2:30–4:30 Presentations on Section Strengths, Weaknesses, and Recommendations and Discussion 4:30–5:00 Synthesis and Closure* *A half-day closing should be considered if the goal of the session is not only to synthesize previous sessions, but also to prioritize and plan interventions. 12 Suggested section pairings and guidelines for group work are provided in Box 3. Some questions in each section can be satisfactorily completed prior to the session, reducing the time needed to complete the remaining questions. However, if this is not possible, follow-up discussions about specific questions will likely be necessary. Box 3. Suggested Pairings and Guidelines for Group Work It is effective to establish six work groups based on the LSAT components—  Group I: Logistic Management Information System (LMIS)  Group II: Forecasting, Procurement, Product Selection  Group III: Inventory Control, Product Use  Group IV: Warehousing and Storage, Transportation, and Distribution  Group V: Organizational Support  Group VI: Finance, Donor Coordination, and Commodity Security Each group should— 1. Choose a group facilitator and presenter. 2. Complete the relevant questionnaire sections for the components assigned to your group. 3. Identify relevant issues to be discussed with the group for validation, if any. 4. Determine at least three strengths and three weaknesses. 5. Provide associated recommendations for each section (see below). 6. Write group work results on a flip chart or create a PowerPoint presentation. 7. Present the group work. Groups should base the recommendations they make on SMART-I parameters, meaning they are:  S = Specific/Strategic  M = Measurable  A = Attainable  R= Realistic  T = Time-bound  I = Implementable Option 2: Key Informant Interviews Set up interviews with key informants and experts to cover each of the LSAT components. During the interviews, review the relevant sections with them and record their responses, asking for supporting documentation when needed. If respondents are unable to answer a question, ask them where the information you are looking for might be found. Present the information collected through key informant interviews in a meeting with stakeholders; you can discuss findings and their implications. The facilitator or interviewer will also need to compile the results in a report. The collected information should identify the key strengths and weaknesses of the system. Using the criteria described in the analysis section below to identify objectives, it should also lead to the development of recommendations and a work plan. 13 Option 3: Discussion Groups and Key Informant Interviews The above-described data collection principles for group discussions and key informant interviews should be employed in tandem when combining both methods in the same assessment.\ Scoring The scoring sheet is an effective mechanism for synthesizing data into a manageable number of questions and, taken together, paint an overall picture of the logistics system. The scoring sheet contains core questions for all 11 sections of the LSAT; instructions on scoring; and summary boxes for strengths, weaknesses, and general recommendations. The scoring sheet can be found in the LSAT document that is part of the toolkit. Data Analysis The information collected using the LSAT can be used to inform the work planning process and can be used to monitor progress over time. Work Planning To inform work planning, users can review the strengths and weaknesses of a logistics system and use the information to develop appropriate objectives and interventions as part of an effective work plan. If time allows, a participatory analysis of the LSAT discussion results is highly recommended. This is especially recommended when group discussions are used for data collection because the participants are already together. Such a session can take up to one day and includes—  Developing a consolidated summary of the key points and observations (e.g., strengths and weaknesses).  If an LSAT has previously been done, comparing the current and prior year’s findings and noting reasons for any significant changes, including assumptions that did not work.  Identifying key existing conditions or circumstances (i.e., the context) that will influence the choice of objectives and interventions.  Identifying objectives or reevaluating objectives from the previous year.  Using the Objectives and Interventions Worksheet (available in the LSAT document) to define and prioritize follow-up actions. Monitoring Commodity security is increasingly a global concern. At the same time as resource availability is decreasing, people’s awareness of and desire to use certain commodities is increasing. Supply often fails to meet demand in these circumstances. Proper management of health products when they are received, and ensuring that they reach the end users for whom they are intended are key elements in meeting the challenge of commodity security. Ideally, to monitor changes in the logistics system over time, the LSAT should be carried out at regular intervals, preferably once a year. Scores from one year to the next can then be compared to track progress and adjust interventions as needed. 14 Box 4. Chapter Summary: Using the LSAT  The Logistics System Assessment Tool (LSAT) is used to assess 11 elements that comprise public and private sector logistics systems, and their enabling environments: organization and staffing; logistics management information systems; product selection; forecasting; procurement; inventory control procedures; warehousing and storage; transport and distribution; organizational support; product use; and finance, donor coordination, and commodity security planning.  Data collection with the LSAT can be conducted through discussion groups, key informant interviews, or a combination of the two. Background research and field visits are key additional activities that help assessment teams adapt the LSAT to better fit the context of the system being evaluated and to obtain information that could not be gained through discussions or interviews.  It is highly recommended that analysis of the LSAT discussion results be conducted with participants, including developing a consolidated summary of key points and observations; comparing results to any previous assessments; identifying existing circumstances that will influence the choice of objectives and interventions; and identifying and prioritizing objectives.  To monitor changes in the logistics system over time, the LSAT should ideally be carried out at regular intervals, preferably once a year. 15 16 Using the LIAT The Logistics Indicators Assessment Tool (LIAT) is a quantitative data collection instrument that is used to conduct surveys that assess health commodity logistics system performance and commodity availability at health facilities. The LIAT can be used to monitor the performance of certain processes involved in the logistics management of health commodities over time, to evaluate certain outcomes of logistics interventions, to provide ongoing supervision and performance monitoring, and to monitor commodity availability. The data collected using the LIAT can be used to calculate the following core logistics indicators:  Accuracy of logistics data for inventory management.  Percentage of facilities that receive the quantity of products ordered.  Percentage of facilities that maintain acceptable storage conditions.  Percentage of facilities whose stock levels ensure near-term product availability (stock status).  Percentage of facilities that experienced a stockout at any point during a given period or at the time of the visit. In addition to these indicators, the data collected can be used to calculate additional related indicators, such as frequency of stockouts over a given period, duration of stockouts, and more. Supplemental questions provide additional information about the characteristics of the supply chain being assessed, such as the use of LMIS information, ordering procedures, transport systems, supervision frequency, cold chain management, and others. Survey Design Experimental and non-experimental survey designs may be used in evaluations conducted with the LIAT. Selecting a survey design depends greatly on what is being measured and the availability of resources for implementation. Experimental Design Surveys that employ experimental design produce the most reliable results. Experimental design comprises data collection from a randomly selected control group and a randomly selected intervention group. This helps establish a fairly clear picture of a program’s impact by comparing the difference between areas where specific interventions have been implemented and areas where they have not. Although experimental design is considered to be the most sound assessment method, it is also the most costly because it requires collection from a large number of sites. In the particular case of supply chain assessments, experimental design may not be feasible because it is often impossible to identify a control group. That is, because supply chains generally cover entire countries, a control group within a country (i.e., an area not served by the supply chain) usually doesn’t exist. With this in mind, experimental design could be very effective for evaluation of a pilot project occurring in a limited intervention zone or to analyze a situation in which different logistics processes are being used in different areas of a country. However, ethical issues of 17 providing an intervention to one group and not to the control group should be weighed before considering an experimental design. Non-Experimental Design Non-experimental designs, which do not include a control group, are widely used by programs that do not have sufficient resources to fund data collection from both an intervention group and a control group. Inasmuch as non-experimental designs lack a control against which to measure changes, they are less reliable than experimental designs. Yet, well-conducted surveys using non- experimental design can still produce valuable, accurate data for decision making in supply chain management, particularly when complementary qualitative information is gathered using the LSAT to determine what factors may have influenced outcomes. Because non-experimental design does not include data collection from a control group, it is important to collect information at the beginning of a program cycle so that comparisons can be made to data collected at later points, especially the end of the program cycle. That being said, collecting data only at cycle end is extremely common in supply chain programs in the developing world, mainly due to limits in funding availability. End-point-only assessments can be useful for understanding the situation on the ground at a particular point, but are not effective in measuring change, particularly when a systems assessment is not undertaken (with the LSAT) at the same time. Programs lacking adequate funding to conduct a baseline survey may choose to carry out initial data collection at a later point when funding becomes available. A program will need to assess whether there is enough time left in its lifecycle to permit the intervention to exhibit meaningful change; since there are many external factors determining the success of a particular intervention, it is up to the program to decide whether enough time has passed to allow the intervention to take hold. Selecting a Survey Design Table 1 summarizes the relative strengths and weakness of the survey design types detailed above. Table 1. Strengths and Weaknesses of Commonly Used Facility Survey Designs Design Advantages Disadvantages Experimental Design Pretest and post-test data collection from control and intervention groups.  Strongest design  Provides a good estimate of a program’s impact.  Data is highly reliable.  Requires that control and intervention groups be similar to one another at start of intervention.  Control group must not be influenced by intervention for length of program cycle.  Cost and time resources can be high. Non-Experimental Designs Pretest and post-test data collection from an intervention group.  Provides an approximate estimate of a program’s impact.  Costs are lower than experimental design approaches because the sample size is usually smaller.  Results are not as rigorous as those in experimental design surveys. 18 Design Advantages Disadvantages Post-test data collection from an intervention group.  Useful for exploratory studies.  Uses the fewest resources.  Weakest design of all survey types because no comparisons can be drawn or impact determined.  Results have the least reliability of all survey types. In cases where a pretest assessment is not conducted, a post-test could still be useful for exploratory studies. However (as noted in Table 1), no comparisons can be drawn or impact determined in this case. In addition to the information provided above, a flowchart for determining survey design type for facility-level assessments using the LIAT can be found in Appendix 3. Sampling After selecting a survey design, the sample from which to collect data must be determined. This comprises identifying the number of facilities needed to produce sufficient information on supply chain efficiency, randomly selecting specific facilities in a manner that produces the right mix of facility types, and developing a strategy to organize collected data. Geographic Coverage It is important to consider the areas of a country that a survey will cover. A large, well-funded survey may include all main regions or states of interest. But a smaller data collection area may be necessary due to limited resources or other programmatic reasons, such as avoiding locations that are not within a program’s reach (e.g., mountainous regions would not be included in a malaria initiative). In addition, accessibility to specific regions may be affected by seasonal variables, such as a heavy rain or political conflict due to approaching elections. Faced with obstacles of this sort, stakeholders will have to decide whether to postpone or scale back an assessment. Whatever the makeup of a geographic area being studied, these decisions will have to be discussed ahead of time with stakeholders during the process of obtaining the personnel and financial resources that are needed for an assessment. The decision to scale back the scope of an assessment must be balanced against the need for sufficient data. An assessment scope that is too limited runs the risk of producing results that may not be representative of the true situation on the ground. Facility Types Depending on the supply chain being assessed, a survey may need to include public, private, commercial, and NGO health facilities. The consideration of which types of facilities to include may also depend on the interest of the stakeholders involved and the feasibility of evaluating the different sectors. It is also particularly important to determine the types of participating facilities before developing the sample size and selecting sites. In the majority of health facility assessments, the lowest level of the supply chain (i.e., the service delivery point) is of most interest to programs. Developing a Sampling Frame Determining geographic coverage and facility types is essential to developing a sampling frame from which to determine the sample size and the specific facilities to visit. A sampling frame is simply a 19 finalized list of facilities that meet the specific criteria of the assessment. Sampling frames should only include those facilities that will be useful in evaluating the supply chain system. The first step in developing a sampling frame is to obtain an updated and complete list of health facilities that are being supplied by the logistics system under evaluation. The list can be exhaustive and include all health facilities covered by the logistics system, or it can be streamlined to include only particular geographic areas. Apart from inclusion in a control group (if an experimental design is being employed), any facilities in geographic areas that are not selected for evaluation due to political, programmatic, or accessibility reasons should also be removed. The next step is to remove from the sampling frame facilities whose administrative characteristics do not fit the purpose of the assessment. Most health facility assessments will look at the various levels of a health system, including primary, secondary, and tertiary health facilities. Several types of health facilities categorized as primary health facilities may not fit the criteria of the assessment due to special characteristics that are not consistent with those of most other facilities. For example, certain health posts may not receive supplies directly from a higher-level facility or may not be required to keep stock records because the number of clients they receive is so high that all available staff time must be devoted to providing services. In such instances, it is practical to remove all such health posts from the facility list since the data from these facilities may not be useful in analyzing the supply chain. In fact, all facilities should be judged in this way to include only those that meet the criteria of the assessment. The final step in developing a sampling frame is to remove all facilities that are known to be closed (permanently or seasonally) or otherwise inaccessible to the data collection teams. Whether removing facilities because they do not fit the criteria of the assessment or they are non­ functional, decreasing the sampling frame must be done in a purposeful manner to avoid the appearance of bias. It is not legitimate to remove health facilities that are remote or are perceived to be unimportant because such facilities often experience the greatest logistics challenges, and excluding them from the sample can give an inaccurate picture of how well a logistic system is functioning. Every viable health facility should have an equal chance of being selected for the assessment and, therefore, should be included in the sampling frame. Calculating Sample Size Ideally, an evaluation team would gather information from the entire group or population of health facilities being targeted in an assessment. But, since this would exhaust available resources in most settings, a subset or sample that allows a fairly accurate representation of all facilities can be assessed instead. For example, a population might include all health facilities in a country’s tropical subregion that provide antimalarial medicines, and a representative sample of that population would consist of five service delivery points at each level of the health system. While visits should be made to as many facilities as possible at all levels of the system, it is not mandatory in all cases to select a statistically significant sample when using the LIAT. This is particularly true for initial program cycle assessments aimed at identifying systemic issues; coupled with information gathered from a LSAT, a non-statistical sample would likely yield valuable information for program managers. However, when establishing baseline data for a program cycle, it is important to have a ample number of facility visits to measure change. Seeking a statistically significant sample is highly encouraged when programs will be called upon to show results at the end of the program cycle or if a nationally representative sample of the health facilities is desired. As demonstrated in the following section, ensuring a sample of high statistical significance can often 20 involve a high number of site visits, although there are steps to take to make a representative sample more attainable. Acquiring Sample Sizes of Statistical Significance To better ensure that the results of any assessment can be considered representative and true (not due simply to chance), evaluators should collect data from a random and sufficiently large sample. Sample size should be determined based on indicators that are of most importance to the program (e.g., in the case of a LIAT, availability of a particular product on the day of visit). It is critical to ensure that a sufficient number of facilities where the particular product is suppose to be stocked are sampled to accurately measure change in the key indicators and confidently present data to decision makers. It is therefore essential to properly define the study population when developing the sampling frame. The three primary factors that are used to determine the sample size for simple random samples are the estimated prevalence of your disease or outcome of interest (i.e., percent of facilities that experienced a stockout at any point during a given period), the margin of error, and confidence level that evaluators determine to be acceptable. The margin of error measures the precision with which an estimate from a single sample approximates the population. The confidence level is the probability that a population estimate lies within a given margin of error. Take the example of an assessment team that sets a 6 percent margin of error (+/-3 percent) and a 95 percent confidence level. If 77 percent of facilities surveyed in their assessment were shown to be correctly using stock cards, one could conclude there is a 95 percent likelihood that 74 to 80 percent of all facilities are correctly using stock cards. In many situations, the margin of error and confidence level may be relaxed to allow for an attainable sample size. A more realistic margin of error and confidence level for a LIAT survey might be 20 percent (+/-10 percent) and 90 percent, respectively. For generating representative samples for a LIAT survey, it is recommended that evaluators set a margin of error at or below 20 percent and a confidence level at or above 90 percent. (Note: For preserving statistical significance, it is more important to maintain a higher confidence level than it is to maintain a smaller margin of error.). Whatever the margin of error and confidence level selected, be sure to clearly state the parameters used to calculate sample size when presenting LIAT survey results to stakeholders. The general formula for calculating a sample size is: n = t² * p (1-p) m² where: n = required sample size t = the value of the confidence level you have chosen (at 80 percent t = 1.28, 90 percent = 1.64, 95 percent t = 1.96) p = estimated prevalence of the indicator. (The product of p and [1-p] is maximized when p = 0.5. Therefore, when prevalence is unknown, 0.5 should be used.) m = margin of error you wish to allow in estimating the prevalence, expressed as a decimal (at 20 percent m = 0.2, at 10 percent m = 0.1, at 5 percent m = 0.05). 21 However, where there is a predetermined population (e.g., total number of facilities in the country), the sample size generated from the above equation needs to be multiplied by the Finite Population Correction (FPC) factor. For our purposes, the formula can be expressed as: New n = n_____ 1+[(n-1)/N] Where: New n = the adjusted new sample size N = the population size n = the sample size obtained from the general formula Using the example above, ensuring a 6 percent margin of error and a 95 percent confidence level would require: Visits to 175 facilities in a population of 500 facilities Visits to 211 facilities in a population of 1,000 facilities However, by expanding the margin of error to 10 percent and a confidence level of 90 percent, it would require: Visits to 60 facilities in a population of 500 facilities Visits to 64 facilities in a population of 1,000 facilities Note that the sample size does not vary considerably between population sizes. The main difference exists between margin of error and confidence levels. The sample size equations given above are for simple random sampling or, in other words, when it is possible to select the respondent (e.g., health facility) from the entire population (e.g., a country). However, due to travel or resource constraints (e.g., logistics, funding), it might be necessary to use cluster sampling to obtain results, where the total population is divided into smaller groups (or clusters) and a sample of the groups is selected. For example, using the study sampling frame developed earlier, the surveyors would begin by randomly selecting at a high level, such as sampling from regions or districts. Then they would randomly select clusters from within those districts/regions, and then within those randomly sample facilities. In these situations, in order to produce survey estimates with the same precision as in a simple random sample, the sample size should be multiplied by the design effect (DEFF) The DEFF can be interpreted as the factor by which the sample size for a cluster sample would have to be increased to produce survey estimates with the same precision as a simple random sample. It reflects the effects of stratification, stages of selection, and degree of clustering used in the facility survey. The magnitude of the DEFF depends on the intraclass correlation coefficient or ICC (i.e., the degree of similarity within the cluster on the variable of interest) and the average cluster size. Ideally, an estimate of DEFF for the indicators can be obtained from prior surveys in any setting. 22 However, when no information is available, the default value of DEFF = 1.2 is recommended with facility-based assessment.1 Thus, the sample size from the above equations is multiplied by the DEFF (1.2) and then divided by the number of clusters to be sampled to determine the number of observations needed per sampled cluster (or district): n per cluster sampled = (New n * DEFF) Number of clusters sampled It should be noted, that in most LIAT surveys one does not need to account for the DEFF, since simple random sampling or probability proportionate to size sampling, rather than cluster sampling, is being used in the sample frame. However, as mentioned earlier, it is very important to base the study sample size on the lowest level of interest. For instance, using the formulas above, one is only obtaining a sample large enough to test for significance at the largest sample level (i.e., national level). If one wants to measure results at the lower levels (i.e., regional or district level), one would need to calculate the sample size at these levels and adjust the total sample size upward accordingly for national representation. To summarize, assessment teams should aim to strike a balance between a realistically feasible number of site visits (given time and resource constraints) and the amount of data that is needed to develop evidence-based, compelling recommendations for improvements to the supply chain. Always consult a statistician before determining the number of facilities to target in a given population. Box 5. Examples of Acquiring Sample Sizes of Statistical Significance Example 1: You are conducting a LIAT survey and have agreed with stakeholders that your survey size will have a 90 percent confidence level and a 6 percent margin of error. While there are 1,800 facilities in 85 districts throughout the 13 provinces in the country, it is only feasible to go to 35 districts where there are a total of 760 facilities. Step 1: Calculating general sample size N = 1.642 * 0.5(1-0.5) 0.062 n = 187 Step 2: Accounting for the FPC New n = 187 1+((187-1)/760) New n = 150 Stop here if using simple random sampling Step 3: Multiplying by the DEFF and dividing by clusters (150*1.2)/35 = 5‒6 facilities per district (for a total of 180 facilities) 1 This value is based on recommendations from the Measure Evaluation “Sampling Manual for Facility Surveys for Population, Maternal Health, Child Health and STD Programs in Developing Countries” (http://www.cpc.unc.edu/measure/publications/pdf/ms-01-03.pdf). For household- or population-based surveys, the DEFF value generally varies between 1.5 and 2.0. 23 Example 2: You are conducting a LIAT survey and have agreed with stakeholders that your survey size will have an 80 percent confidence level and a 5 percent margin of error. While there are 800 facilities in the country in 10 districts, it is only feasible to go to 40 districts where there are a total of 200 facilities. Step 1: Calculating general sample size N = 1.282 * 0.5(1-0.5) 0.052 n = 163 Step 2: Accounting for the FPC: New n = 163 1+[(163-1)]/200) New n = 90 Stop here if using simple random sampling Step 3: Multiplying by the DEFF and dividing by clusters (90*1.2)/40 = 2‒3 facilities per district (for a total of 108 facilities) . Deciding upon a Sampling Methodology There are two types of sampling methodologies: probability sampling and non-probability sampling. Probability sampling involves randomly selecting members of a population to create a sample, thereby reducing the risk of bias because each member of the population has a non-zero chance of being included in the sample. Non-probability sampling relies on the judgment of the assessment coordinator to select the study participants. Probability sampling should be employed in assessments that use the LIAT, while non-probability sampling is more appropriate for qualitative evaluations, including those conducted with the LSAT. Types of Probability Sampling Below are four commonly used types of probability sampling: simple random sampling, stratified random sampling, systematic sampling, and cluster sampling. A flowchart for determining sampling method for facility-level assessments using the LIAT is provided in Appendix 4. Simple Random Sampling Simple random sampling begins with the creation of a sampling frame that includes all eligible facilities. Facilities in the sampling frame may be identified by name or by an assigned number. Then, using a random number generator (see “Steps in Sampling” below), facilities are selected one by one until the number of selected facilities equals the size that was determined for the sample. It is important that sampling be done with replacement. That is, all of the facilities (even those already selected) should be included in the sampling frame each time a new random selection is made. This allows all facilities to have an equal chance of being selected and simply requires that, when a facility is selected more than once, it should be skipped, and the random selection process should continue until the desired number of different facilities is reached. Simple random sampling is easy to use and is considered to be an unbiased selection process. However, simple random sampling does not allow for sampling of different subgroups (strata) within a population. For example, hospitals and primary health facilities would have an equal chance 24 of selection in simple random sampling, so the resulting sample could have an extremely high number of one type of facility and very few of the other. Stratified Random Sampling Stratified random sampling has the same characteristics as simple random sampling, but it allows for the analysis of different strata within a population. Simple random sampling is carried out for each subgroup instead of only for the population as a whole. For example, separate samples of warehouses, hospitals, and primary health facilities would be established. In this case, a sample size of each stratum would need to be calculated. The benefit of stratified random sampling is its ability to facilitate meaningful comparisons between strata. Strata can be defined as the different levels of a supply chain, different facility types, or different ownership of the facilities (i.e., public versus private). A challenge exists in appropriately defining the different strata needed for the desired comparisons. The central issue with stratified random sampling is that the cost and complexity of the assessment tend to increase each time a new stratum is defined. Systematic Sampling (Proportional-to-Size Sampling) Systematic sampling is a third sampling technique that can be used to select facilities for an assessment. This sampling technique uses a sampling interval to randomly select facilities from the sampling frame. The sampling interval is calculated by dividing the total number of facilities in the sampling frame by the number of facilities determined for the sample size. For example, a sampling frame of 1,000 and a sample size of 100 would yield a sample interval of 10. With this figure, a starting point is selected at random using a number between 1 and 10, and then every 10th facility following the random number starting point would be selected until the number of facilities selected equals the sample size. Systematic sampling is easy to use, especially when dealing with a large number of facilities in a complete sampling frame. The disadvantage of using this method is that bias could be introduced if the list of facilities is patterned in such a way that it promotes the selection of a certain type of facility. For example, the list is organized by facility type so that every 10th facility is more likely to be a hospital than it is to be a service delivery point. Cluster Sampling A cluster is a naturally occurring subgroup, such as health facilities under an administrative area (e.g., region, district). In cluster sampling, all or a selection of health facilities are chosen from a select number of subgroups. Cluster sampling uses a multi-stage approach where subgroups are selected from the various levels of a supply chain and, finally, from health facilities within the cluster. This sampling approach is particularly useful for assessments with limited resources; fewer geographic areas need to be visited, which helps to reduce overall costs. However, considerations with regard to sample size must be addressed, particularly, the “design effect” where the number of facilities is multiplied to account for differences between clusters. A main difference between stratified random sampling and cluster sampling is that stratified random sampling involves selecting facilities from the whole population of the strata, whereas with cluster sampling, selection occurs only within a limited number of selected clusters. 25 Comparing Probability Sampling Methods Table 2 summarizes commonly used probability sampling types, including the four types described above for surveys that employ the LIAT. Table 2. Commonly Used Probability Sampling Types Sampling Type Description Benefits Issues Simple Random Sampling Each unit has an equal chance of selection. Unbiased; relatively easy to use. Groups of interest may not be selected in proper proportion. Stratified Random Sampling Population is divided into meaningful groups or strata. Enables analysis of subgroups. Sample size must be calculated for each stratum; can be costly if using many strata. Systematic Sampling (Proportional-to-Size Sampling) Every Nth unit on a list is selected based on dividing the size of the population by the sample size. Convenient if facility list is provided; as good as random sampling if starting point is obtained randomly. Attention must be paid to recurring patterns within the sampling frame. Cluster Sampling Sampling is conducted from naturally occurring subgroups. Sampling from subgroups may allow for savings in time and logistics. Design effect may necessitate an increase in sample size. Using Purposeful Sampling for LIAT Because most health commodity logistics systems enable health products to reach service delivery points via a system of warehouses or depots, it is important to assess these storage points. Some LIAT indicators are designed to enable evaluation of the linkage between service levels, including region-to-district and district-to-service delivery point. Thus, a survey often needs to employ a certain amount of “purposeful sampling” to guarantee that the facilities selected for the sample are truly linked to each other. Alternatively, random sampling can be done at a higher level (regional level) and, then further sampling can be done at the district level before sampling the health facilities within that district. Attention will have to be paid to whether the selected districts have an adequate number of health facilities to properly reach the required sample size of health facilities. Steps in Sampling Once a sampling frame and sample size have been generated, a representative sample of health facilities can be drawn using the sampling methodologies mentioned above. As long as a sample is ultimately determined without bias, any type of random number generator can be used, including rolling dice, drawing numbers from a hat, using a random number table, or using a website designed to generate random numbers (e.g., http://www.random.org). Example: Generating a Sample Using Simple Random Sampling The Ministry of Health in Sangala wanted to conduct an assessment of its malaria supply chain, but it did not have the resources to visit every service delivery point that offers antimalarial medicines. In consulting with a statistician from the country’s National Statistical Service, the Ministry decided that the most efficient way to conduct its assessment would be to generate a random sample that is representative of all facilities offering antimalarials. 26 The Ministry then: 1. Established a complete list of health facilities providing antimalarials, including verifying the geographical coordinates that were collected for each during an earlier assessment. 2. Excluded all facilities that did not fit the scope of the assessment, including privately operated clinics and facilities known to be closed. 3. Numbered the health facilities from 1 to 20, shown in the sample list below. (In a real-life situation, sampling would not usually be conducted on a sampling frame of only 20 facilities; instead, all of the facilities would be visited. Sampling should be considered when the total number of facilities in a sampling frame is too great to visit.) Sangala—Sampling Frame for Antimalarial Medicine Supply Chain Province District Facility Name Order Number Southern Balaka Kipiri Heath Center 1 Southern Balaka Balaka Health Center 2 Southern Balaka Chilipa Health Center 3 Southern Balaka Chyendu Health Center 4 Southern Balaka Confort Health Center 5 Northern Chitipa Chambo Health Center 6 Northern Chitipa Ifumbo Health Center 7 Northern Chitipa Kameme Health Center 8 Northern Chitipa Kapenda Health Center 9 Northern Chitipa Kaseye Health Center 10 Eastern Dedza Bembeke Health Center 11 Eastern Dedza Chikuse Health Center 12 Eastern Dedza Chimoto Health Center 13 Eastern Dedza Chipwanya Health Center 14 Eastern Dedza Chitowo Health Center 15 Western Dowa Bowe Health Center 16 Western Dowa Chakhaza Health Center 17 Western Dowa Chankhungu Health Center 18 Western Dowa Chinkhwiri Health Center 19 Western Dowa Chisepo Health Center 20 4. Used a random number generator to select five service delivery points, which was the sample size that had been calculated for this assessment. The Ministry was careful to include all 20 numbers each time a site was to be selected using the random number generator and, on two occasions, numbers that had already been selected were redrawn and had to be skipped. The resulting facilities to be visited were those corresponding to numbers 3, 6, 11, 14, and 18. 5. Followed up the sample selection by ensuring that the facilities identified for visits were still operational and would be accessible to the data collection teams. 6. Randomly selected two additional facilities (i.e., 10 percent of the total list) to act as replacements if facilities in the sample were not accessible. The Ministry was careful to select 27 these replacement facilities ahead of time to reduce the appearance of bias in the sampling process and to ensure that replacement facilities met the criteria of the assessment. Establishing a Unique Identifier When the sampling process is complete and the number of health facilities selected is equal to the sample size that was set, a list of the selected facilities can be developed. The key to making this list an essential tool for data organization is assigning a unique identifier for each facility. The unique identifier is typically a numbering scheme that follows a logical progression and makes every facility distinctive. For field work, this unique identifier is important because it allows data collectors to track various forms related to a particular facility. Each form and page of the survey instrument should be marked with the facility’s unique identifier, thus facilitating the submission of completed surveys at the end of data collection, as well as data entry and management. Unique identifiers (as opposed to facility names or geographic coordinates) also provide facilities with a relative amount of anonymity during the data analysis phase. Although a unique identifier can be any alpha-numeric sequence, a particular numbering scheme is recommended for assessments using the LIAT. The main reason to use the recommended numbering scheme is that it matches the database protocols that were developed to accompany the LIAT and this document. The unique identifier should be made up of three sequences of a three- digit number: ###.###.###. The first three numbers correspond to the code given to the state/region being assessed, the next three numbers correspond to the code given to the district or local government area, and the last three numbers are the code for the facility. Table 3 shows an example of how this numbering scheme for the unique identifier might work. Table 3. Example of the Facility Unique Identifier Number Scheme Region Name Region Code District Name District Code Facility Name Facility Code Unique Identifier Luapula 111 Manza 111 Manza HC 001 111111001 Luapula 111 Manza 111 Manza Hospital 002 111111002 Luapula 111 Mweru 222 Mweru HC 003 111222003 Eastern 222 Chipata 333 Chipata HC 004 222333004 Eastern 222 Kazembe 444 Kazembe Hospital 005 222444005 Western 333 Mongu 555 Mongu HC 006 333555006 In many countries, health facilities may already have a unique identifier assigned to them. Even in these instances, it is recommended that the numbering scheme described above be used for surveys, since many facility codes are long or are made up of a completely randomized set of numbers. Still, local facility codes may assist in identifying a facility at a later stage in a survey, so the LIAT data entry tool has a space to record them. 28 Survey Preparation Once an agreement between stakeholders has been reached to conduct a supply chain assessment, and adequate resources have been secured, a significant amount of planning must occur before data collection can begin. This includes staffing, selecting data collectors, obtaining authorization to visit facilities, organizing survey logistics (e.g., transportation to sites), and preparing materials. Staffing Assessments generally include the participation of:  trainers;  data collectors, including team leaders;  a survey coordinator;  survey monitors (optional);  a data analyst (optional); and  data entry staff. Determining personnel requirements is a critical step in ensuring that an assessment stays within its established timeline and budget. Depending on the size of an assessment, it may be necessary for program staff who are familiar with the technical area being assessed or who have prior experience in quantitative assessments to be part of the team. Local team members are also helpful in identifying data collectors. Often, outside data collectors can be recruited from a local university, ministry of health, or nongovernmental organization. A data entry staffer may also be needed and can be outsourced if the program staff is unavailable for this task. Additionally, data collectors may function as team leaders in the field to help with daily quality control checks (explained in detail below) and general coordination. Team leaders can be selected from the pool of data collectors based on their understanding of the assessment tool and data collection protocols. A good time to assess the qualifications of team leaders is during piloting of the assessment tool (also explained below) at practice facilities. If local program staff or external technical assistance is not obtained for the assessment, a locally hired survey coordinator will be needed. Survey coordinators are contracted to oversee the entire data collection process and to provide guidance to the data collection teams. Survey coordinators will need to be supervised by the person in charge of the activity or the technical lead. Survey coordinators may also be tasked with analyzing data and reporting the results of the study. In some cases, it may be necessary to contract separately for a data analyst to assist with analyzing the data. Sample scopes of work for both the survey coordinator and data analyst positions are provided in Appendix 5. Selection of Data Collectors Selecting who will administer the survey is important to the success of the activity. A person’s background in health, previous experience with surveys, familiarity with the geographic area in question, and ability to speak the local language are all important considerations. Familiarity with supply chain management and the health area being evaluated is often the primary factor in choosing data collectors. Program personnel who have experience with program monitoring and evaluation (M&E) activities and who understand internal administrative and financial rules will also facilitate the progress of an assessment. 29 The data collection team should include personnel from the ministry of health (MOH) involved in implementation of the logistics system. Inclusion of MOH staff will facilitate the data collection process by granting a team access to key local government personnel and using their knowledge of government processes. It is also quite valuable to have MOH personnel involved in the data collection process so they can get a first-person look at the situation in the field. This may aid a program’s efforts in advocating for change based on assessment results. For particularly large surveys, staff outside the program or MOH may need to be recruited. Students or faculty at local universities or teaching hospitals make good candidates, especially if they are involved in public health. Sample job descriptions for the recruitment of data collectors and data entry staff are provided in Appendix 6. Regardless of their background or affiliation, it is critical that data collectors be available and can commit to the entire data collection period. Prior experience on other surveys does not eliminate the need for data collection training, which is specific to each survey. It is useful to have potential substitute data collectors attend the training to ensure that last-minute personnel substitutions can be made if the need arises. The steps to take when acquiring survey staff are: 1. Determine local staff resources and the number of data collectors needed for the survey, and choose whether a survey coordinator and data analyst external to the program should be contracted. 2. Advertise for and select a survey coordinator (and data analyst, if needed) for the entire survey. 3. Advertise for data collectors. Plan to hire no more than three to four data collectors per team with the expectation that each team will visit an average of two facilities per day. The number of teams required will depend on the overall number and geographic distribution of the facilities and time dedicated to data collection. 4. Develop contracts and statements of work (SOWs) for all external survey staff, and ensure that external contractors understand their commitment for the length of the survey process. Planning for Survey Logistics It is essential to have a designated vehicle and driver for each data collection team. Multiple vehicles may have to be hired for a large survey. Borrowing vehicles from outside projects or a ministry of health may seem like a good way to save money, but partner agencies’ priorities can change and may result in a sudden requirement to return a vehicle. Vehicles must be committed for the entire duration of the data collection phase, including contingency plans in the case of breakdowns and funds for fuel. Planning a field visit itinerary that ensures the least distance to travel between facilities and appropriate areas for evening lodging (e.g., midway between the last facility to be visited on one day and the first to be visited on the subsequent day) is an important way to control expenses and maximize the use of time spent in the field. With this in mind, a schedule should be determined ahead of time so that each data collection team knows which location to cover at the beginning of any given day and where it will need to be after data collection is completed for that day. The first day of data collection is usually spent visiting the Regional Medical Officers to ensure that they are well aware of the data collection efforts occurring in their area. District Medical Officers should also be visited before beginning data collection in each new district. This protocol visit will further facilitate cooperation from the health facility personnel. Identifying appropriate places to obtain fuel and meals should also be considered when planning a day’s activity. 30 The following steps should be taken when arranging survey logistics:  Make sure that the location of each facility that has been selected to participate in the survey is known, including how to reach it (what roads to take). Prepare a draft schedule of field visits for each team.  Arrange lodging for the data collectors; alternatively, data collectors can make arrangements themselves.  Arrange for questionnaires to be returned to the survey coordinator.  Contact local government entities, as appropriate, to inform them that a survey will take place.  Obtain letters of introduction from the MOH for the data collection teams. Materials Needed for Training and Field Work In addition to paper and pens, there are a number of items that will need to be secured before an assessment begins. A complete list of materials needed for survey training and field work is provided in Appendix 7. Computers should be designated or rented for data entry and analysis, including one computer for each person entering and analyzing data. The computers should be equipped with software that is compatible with the needs of the assessment (e.g., Microsoft Access for data entry and SPSS for data analysis). Paper-based surveys may require a significant number of copies, so it is thus important to secure access to an office photocopier or prepare to send out copy jobs to local vendors. In areas where electricity is an issue, it is essential to check whether generator backup is available to avoid delays. A number of electronic devices can assist in the data collection and data entry process. These devices can include rugged laptops, personal data assistants (PDAs), and mobile phone technology that enable data collectors to complete electronic questionnaires. An assessment plan that includes use of these devices should include at least one device per data collection team, although more devices per team would allow data collectors to go to separate locations within a facility and collect data more rapidly. Additionally, if GPS coordinates are being captured for electronic mapping, each survey team should have a GPS device. In some cases, PDAs or cell phones may have GPS capability built in. Each team should have at least one designated mobile phone for communicating with the survey coordinator or country office. In most cases, data collectors will have their personal mobile phone with them in the field, so providing phone cards to team leaders is recommended to encourage them to communicate with the study coordinator and other teams when issues arise. Data Collection Training Training data collectors is a key component of an assessment to ensure complete, high-quality results. This section provides guidance on training data collectors for a LIAT-enabled assessment and should be used in conjunction with the LIAT Trainer’s Guide provided in Appendix 8. Timing Training should be scheduled approximately six months or more in advance and in coordination with all partners involved, including the Ministry of Health (MOH) or host-country government counterpart. The training event itself should take place the week directly before data collection is to 31 begin. Time is a major consideration, as invitations and notification letters sent outside the capital city can take a month or longer to reach their intended destinations. After invitation letters are sent, a follow-up phone call is recommended to confirm participants’ availability. Data collection training generally lasts four days, including a pilot test of the tool. Trainers Trainers should be knowledgeable about the data collection instrument, experienced in the data collection process, and comfortable conducting training. Usually, two trainers are sufficient, but more may be included. Prior training experience is advantageous but not mandatory. When GPS devices are being used to collect the geographic coordinates, one of the trainers should be knowledgeable about GPS and GIS. Participants Data collectors and survey monitors (if used) should be identified well in advance of the training event and data collection activity. It is highly recommended that a national assessment include regional MOH personnel, district MOH personnel, or host-country government staff members who are knowledgeable about the supply chain system and who work in the geographic area where data will be collected. The number of participants will depend on the number of data collection teams required to carry out the assessment. Generally, teams are made up of two data collectors, although some assessments may use up to four data collectors per team so they can further split into two- person teams in the field, as necessary. However, teams comprising four data collectors are not recommended due to the potential for logistical problems and the additional costs this entails. Taking into account budgetary considerations, monitors may travel with a team for the first few days of data collection, or they may visit several teams within a smaller area. Other Participants Generally, senior members of the MOH, host-country government, implementing organization, or primary donor should give welcoming remarks and opening comments. Some of the same stakeholders may participate in the training. It is also necessary that a technical advisor or equivalent provide an overview of the in-country logistics system to improve the participants’ understanding of the data they will be collecting. Data entry personnel who will enter data into a database after the survey can benefit from participating in the training to acquire a greater understanding of the tool and data. They can also serve as data collectors, which will help them better apply their experience with the tool to the task of data entry. Monitors should also participate in training to fully understand the questionnaire and the conditions they will face in the field, even if they have served as monitors in other surveys. Venue When choosing a venue for the training, the following should be taken into consideration:  Proximity to where participants are staying and the implementing organization’s local office.  Whether lunch and tea breaks are included.  Room size in relation to the number of participants.  Whether extras, such as projectors and flip charts, are included in the price of space rental. 32 The training will be much easier on participants and trainers if the venue is located near where they are staying (for those arriving from out of town) and the implementing organization’s local office. Hotels are ideal because they usually have conference rooms, as well as a restaurant for lunch and tea breaks. Holding the training close to the implementer’s office will make it easier to carry training materials, make photocopies, retrieve extra supplies, and complete other administrative tasks. The room should be large enough to accommodate all participants easily, but not so large as to require participants to shout. It will be less expensive if the local office can provide a projector, flip chart, laptop, and any other materials. Materials Handouts and Supplies Materials should be purchased beforehand for all participants (reminder: see Appendix 7 for a sample list of materials needed). All handouts should be printed prior to the training, again before the pilot test, and last, when the tool is finalized before the participants leave for data collection. Participants should be given a binder or folder to keep their handouts organized and in one place. Handouts can either be included in the binder before distribution to participants or they can be handed out daily for the participants to place in their binders. If handed out daily, handouts should be organized in a logical fashion. If draft versions are given out, it is essential to collect and recycle them before the final versions are provided so that no old versions are accidentally used for data collection. Trainers will need to organize revised printed handouts as the training progresses and in time for participants’ use. Products and Forms Samples of products that will be included in the survey should be purchased or borrowed beforehand for the participants to become familiar with packaging, units of count, strength (if applicable), and other details. Examples of stock cards, requisition vouchers, and other logistics forms being used in-country should also be given to participants. Schedule A training schedule is an important tool for training sessions. The training usually lasts four days and is held from 8:30 AM to 4:00 PM (including a one-hour lunch and two tea breaks) each day. An illustrative schedule can be found in Appendix 9. That schedule can be amended, but the sessions should be kept in the order shown because they are purposely arranged for practical reasons. Sessions must start on time and stay within their allotted time or participants will need to stay late to cover all material. It is recommended that this principle be included in the training norms (refer to the trainer’s guide for suggested norms). If an individual or a group of participants arrives late, trainers will need to stay late with them to be sure they understand what they missed. Conducting the Training Trainer’s Guide The trainer’s guide included in the toolkit that this guide introduces serves as a step-by-step tool for conducting training sessions, including all course content. 33 Training Objectives It is important to establish training objectives for participants. The following are examples and can be adapted as necessary: By the end of the training, participants will be able to:  Describe the purpose and objectives of the assessment.  Understand and use the LIAT to collect data on logistics management of health commodities.  Describe the team’s responsibilities in conducting the assessment. Tips for Trainers Trainers should always be prepared and should adhere to the objectives and goals of the course. They should encourage active participation while keeping participants on topic and offering praise and recognition. Giving real examples from past assessments is the best way to enhance participants’ understanding. Trainer Tool Options Trainers may choose use PowerPoint, overhead projectors, or simply speak about content. Using visual aids, such as products and stock cards, is also effective for participant comprehension. Survey Manual The survey manual serves as a resource for data collectors during the training and in the field. It contains an overview, as well as sections on team member requirements, site selection, data organization, ensuring quality, the data collection tool, interview skills, and logistics indicators. A sample survey manual is provided in Appendix 10. Pilot Testing of the Survey Instrument A pilot test will allow data collection teams to practice their newly learned skills in collecting data with the LIAT and, if needed, using GPS devices. The pilot test enables data collectors not only to experience what is likely to occur in the field, but also to identify potential obstacles, issues, and questions prior to real data collection. Pilot sites should be selected, notified by a government official, and confirmed in advance. It is best for them to be in the same area as the training is being held or no more than an hour’s drive away. Each team should visit up to two health facilities and should take no longer than four hours to finish. After the pilot test is complete, teams will return to the training venue and share their experiences and consider revisions to the data collection tool, as deemed necessary. The revisions will result in a final tool that is adapted to the country context. The pilot is also the first time data collector teams will work together. Team members should decide who will carry out each task (e.g., conducting the interview, taking a physical inventory, completing a stock status table). Often, team members take turns to experience each element of the assessment. Data Collection Scenarios Due to the subjective nature of some questions, data collectors will often come across a situation in the field that can be interpreted in different ways or may not be fully covered in the training. It is impossible to think of all scenarios; however, it will be helpful to contemplate these potential obstacles and provide examples during the training. The following real examples could be reviewed on the last day of the training to assist data collectors in how they may answer the question. 34  Storage Conditions  Question: Should products that are stored in the drawer of a staff member’s desk or in a small cabinet at a health facility be assessed using the indicators of storage conditions?  Answer: Yes. Although a drawer or small cabinet is not considered a storeroom, it should still be assessed. Use of the drawer should also be noted in the comments section.  Water  Question: If a health facility purchases water and keeps it for use for hand washing, should this be considered as operational water on the day of the visit?  Answer: Yes. Although it is not from a pipe, well, borehole, or stream, it is still considered an operational way for the health facility to obtain water.  Roads  Question: If the road to a health facility is unpaved and in bad condition, is it considered navigable?  Answer: It depends. Generally, if the team was able to drive up to the facility despite the poor state of the road, it should be considered a navigable road. Team Overview and Mapping for Data Collection In this session, teams will sit together and review site assignments and the data collection schedule. Together they should map out what facilities they will visit on what day, keeping in mind that large facilities will take longer for data collection than will smaller ones. Having a data collector from the area will be helpful in determining the logistics of traveling to facilities that may not be on a map. Team Leader Training Each team should identify a team leader. The team leader will have extra responsibilities to ensure that surveys are complete and carried out according to instructions. Guidelines for team leaders are outlined in the LIAT Survey Manual, and a sample checklist for team leaders can be found in Appendix 11. Trainers should sit with the team leaders to review the checklist and their responsibilities. This should be during the session on data collection details, held on day four of the training. Team leaders are also responsible for labeling envelopes with the unique identification number and province, state, or region for each facility to be visited. After a visit to a facility, the team leaders should place final surveys in the envelope and ensure that they are returned safely to the survey coordinator. Team leaders should also be prepared to debrief with the survey coordinator to discuss the data collection process and any outstanding issues. Monitor Training Similar to team leaders, monitors have extra responsibilities to ensure the quality of surveys. Monitors answer questions teams may have, troubleshoot problems, and ensure that data are being collected according to instructions. The monitor’s checklist can be found in Appendix 11. Trainers should sit with the monitors and review the checklist and their responsibilities. This should also be during the data collection session on day four of the training. Team leader and monitor trainings can be done simultaneously with each trainer. 35 Data Collection Data collection is one of the most important aspects of a health facility survey. The process can take from one week to several weeks, depending upon the sample size, the number of data collection teams, and the distribution of the facilities. Data collectors should have a clear understanding of the objectives of the survey, including why it is being done, by whom, and how it will benefit the health facilities. Although each team is required to visit a set number of health facilities within an established time frame, it is equally important to be careful while collecting data in order to maintain data quality and integrity. Data collection efforts will essentially be worthless if data quality is compromised and the results of the survey are questioned. The sections below describe the process that should be followed to ensure that high-quality data are captured. Organization of Data Collection Teams Data collection teams should be organized so that data can be collected efficiently in terms of the time and resources it takes to conduct surveys and according to the specified data collection schedule. As noted earlier, teams should include at least two data collectors so individuals can alternate between asking survey questions and recording responses. Teams of two data collectors are also advantageous because they require fewer resources and can accommodate the addition of local MOH personnel to the team for facilitating access to health facilities. Teams comprising more than two team members can be useful if facilities in the study are large or have various departments or wards (like hospitals). That is, more ground can be covered in the same amount of time. For example, while two data collectors are conducting an interview, a third can engage the storekeeper to begin a physical count of products, especially if there are many commodities to assess. Teams of more than three individuals are not advisable unless team members can visit two facilities simultaneously. Subdividing a group to cover more facilities can be useful, but it also carries a number of potential pitfalls. If teams are sharing a vehicle, each team will have to wait to be picked up or dropped off at a time when they could otherwise be moving toward another facility. Another drawback to using this “pick and drop” method is the possibility of a vehicle breakdown, stranding one or both teams in unfamiliar surroundings without a vehicle. The best use of two or more additional team members would be to work independently of the first group, in their own designated vehicle, and then meet up in the evenings to discuss survey results. Each team member is responsible for making sure that questionnaires are complete, all answers are clear and reasonable, and hand-written answers or comments are legible. Team members should review the questionnaire for completeness before departing the facility so that missing information can be gathered without having to return to the facility later. It is also highly recommended that the data collectors obtain a functioning telephone number of the facility in-charge or the person interviewed in case the team needs to call the health facility for clarifications or to obtain any information missed during the facility visit. Team Leaders and Survey Monitors Team leaders are data collectors who take on the additional responsibility of checking surveys to make sure that data collected at the facility are recorded completely and accurately. Team leaders may use checklists to ensure that all requirements of the data collection process are being met. At the end of each day, team leaders should lead a review of the data collection process to record any additional information or issues relevant to the study. 36 Team leaders should also make certain that routine administrative processes are carried out to facilitate the data collection process. These may include ensuring that teams arrive at the facilities with ample time to administer the survey, meeting with local government officials upon arrival and departure, providing per diem to local personnel assisting in the study (if appropriate), and maintaining contact with the study coordinator or monitors. Survey monitors add an additional layer of quality assurance to the data collection process. Survey monitors may accompany a data collection team to the field in order to handle questions, troubleshoot any data collection issues, and ensure that data quality checks are being performed daily. Survey monitors typically stay with a team for only the first few days of data collection, but the additional oversight they provide to the overall survey is invaluable. Ideally, there are multiple monitors so that all data collection teams are visited during their first few days in the field. Using survey monitors to their full potential will require additional resources and must be carefully considered for surveys with limited funding. Sample quality checklists for the team leaders or survey monitors are provided in Appendix 11. Survey Submission Process and Call-In Schedule In addition to organizing teams to include team leaders and deploying survey monitors to the field, other practices can be employed during a survey to help ensure data quality. Data collection teams may be instructed to submit completed surveys to the survey coordinator after the first few days of data collection. This way, surveys can be reviewed and any potential data-capturing errors or questions can be identified for immediate follow-up. This process is especially important during the first few days of data collection, when mistakes and or data collection errors can be corrected before too much time and too many resources have been used. Submitting completed surveys for a timely review by the survey coordinator is particularly important when a survey’s budget does not allow for the use of survey monitors. It can also prove to be an inexpensive way to monitor progress in the field once survey monitors are recalled. Data collection teams should submit originals of the completed surveys and retain copies in case the originals are lost. Completed copies can be sent using a reputable courier service. Such service may be offered through the national airlines or print media. Whatever the method of submitting the completed surveys, data collection teams should consider the timeliness of the service; in some instances, it may be better to simply return completed surveys in person at the end of the field work. Before departing for the field, each team should have detailed instructions on how and when it will need to submit completed surveys. Related to submitting surveys for review, telephone calls should be scheduled between the data teams and survey coordinators (or survey monitors). Teams should expect calls from the survey coordinator or monitors on a predetermined date and time. These calls serve as a time to discuss feedback on the previously submitted surveys, relay important instructions on the data collection process, or determine a team’s progress toward visiting the facilities according to the site visit schedule. Call-in may be more frequent during the first week of data collection, especially if an assessment activity chooses not to use survey monitors. Data collectors should be encouraged to call in with questions or concerns at any point during the data collection process. Each team should be supplied with a call-in schedule so calls can happen on schedule and without problems. 37 Data Collection Protocols Completing a survey at a primary health facility usually takes between two and four hours, depending on the size of the facility. When they have planned ahead, teams should be able to visit at least two primary health facilities each day. Hospitals and health facilities at the secondary and tertiary level may take a full day. Team leaders should assist their team in adhering to the site visit schedule to complete the requisite number of facilities in the time allotted. Local Government Briefing Before conducting a survey in a new administrative area (e.g., region or district) the team leader should bring the entire team for a visit with the local government office to brief officials on the assessment activity. Letters of introduction and a list of health facilities to be visited should be shared with officials at this point, and any potential issues should be discussed in detail and resolved prior to beginning data collection. If local government personnel are needed for assisting with the data collection, it is essential to describe how they will be assisting and for how long. Once data collection is completed within this government area, debrief the same government officials to let them know that the team has completed the work and provide them with an overview of the findings for that area. Observations shared with local government officials about visited facilities should remain neutral, touching on both good and bad aspects of the supply chain system being evaluated. It is important to keep individual and facility names confidential as much as possible during debriefings. Replacing an Inaccessible Facility As noted in the sampling section of this guide, some survey teams may encounter facilities that are closed or otherwise inaccessible. In these instances, a decision will have to be made to replace the facility. Replacing the facility can be the responsibility of the study coordinator or the team leader, depending on the organization of the assessment. In most instances, a team leader should contact the study coordinator before selecting a replacement facility. The study coordinator will work with the team leader to find a replacement facility that matches the characteristics of the facility to be replaced. Often, this will be a facility that is in close proximity to the team to minimize the drive time to the replacement facility and avoid falling behind schedule. If a facility replacement list has been established (as described in the Steps in Sample section), team leaders can select the nearest health facility on that list. Team leaders should first try to contact the study coordinator to discuss the decision to replace the facility. If, however, the study coordinator is not available, team leaders should thoroughly document the reasons behind replacing a health facility. Capturing Geographic Coordinates Using Geographic Information Systems (GIS) facilitates the identification of logistics issues that may be specific to certain areas of a country. GIS adds value to information that is obtained with the LIAT by increasing the ways the data can be used; it also provides a simple method for visually demonstrating results, the importance of which is discussed briefly in the last section of this guide. Understanding how geography, both in its human and physical dimensions, affects the functioning of the logistics system will provide decisionmakers and logistics managers with a more comprehensive set of information to assist them in their work. For example, if stockouts in health facilities are observed to be clustered in a particular area, there may be a common cause, such as 38 poor road conditions. Knowing the spatial distribution of logistics indicator values at the health facilities allows for a better understanding of challenges like these and enables targeted decisionmaking and action to improve system efficiency. In assessments where geographic coordinates are being captured, a GPS device or a GPS-enabled device (like a Smartphone) can be used for this purpose. In some cases, geographic coordinates may have already been documented by the client or can be found in previous assessments. Those collecting geographic coordinates during a survey will also require GPS/GIS software to map data. Guidelines for use of GPS devices and GPS/GIS software are included in the toolkit this guide introduces. Data Entry Data entry will begin once surveys are delivered from the field and all necessary checks and corrections have been made to the survey forms. Data entry personnel should be able to work without interruption until the surveys are entered from the data entry tool into the database. In most cases, the completed surveys will arrive from the field intermittently, so data entry staff will likely work for a day or two and then be on call for when the next batches of surveys comes back from the field. The value added of using electronic data-capturing devices such as PDAs or cell phones during the LIAT is that it eliminates the need for data entry staff. Data from electronic surveys can be directly downloaded into a master database, which can then be checked by the survey coordinator or data analyst. Savings in the overall cost and time it takes to enter the data are typically realized here. Data Entry Staff Training Before data entry personnel begin transposing data from the paper-based survey to the data entry tool, a detailed training on data entry protocols and orientation to the data entry tool will be required. Protocols for entering data are described in the Data Entry Guidelines, available as part of this toolkit. Generally, data entry training is quite simple. Its purpose is to orient the staff to data entry procedures and provide them with hands-on practice entering data. Ideally, the data entry staff will have attended the data collector training and perhaps even participated in the data collection process. If, however, the data entry team is unfamiliar with the questionnaire, it is important to orient them to the survey instrument and instruct them on its “skip logic” (i.e., the path that respondents take through a survey based on the answers they give to certain questions and other technical aspects). The survey coordinator or data analyst should sit with the data entry staff as they begin to enter the data to answer questions and provide further instructions as needed. A quality check of the data should be performed early to detect any errors caused by the data entry process. Under the close supervision of the survey coordinator or data analyst, data entry personnel can begin entering completed questionnaires. Alternatively, if practice is warranted before actual completed surveys are available, data collected from the pilot sites can be used as a proxy. In this case, it is essential to erase practice records or create a new database before beginning with completed surveys from the actual survey sites. 39 Data Entry Quality Assurance Having quality data will ensure that the results of the study are reliable and reflective of the true conditions in the field. A number of data quality controls are employed in the data entry tool, such as programmed skip patterns and pop-up reminders. These controls are outlined in the toolkit’s Data Entry Guidelines and are designed to assist the data entry staff in avoiding errors that could undermine the reliability of the results. In addition to these quality control measures, it is strongly recommended that a double data entry model be used to ensure that the data entered are accurate. Double data entry is a quality control measure that involves having two data entry staff entering the same data into two separate databases, and then comparing the databases for discrepancies. If discrepancies between the two databases are identified, the response from the questionnaire is manually rechecked and the change is made to ensure that correct information appears in both databases. A Microsoft Excel document designed to check for discrepancies between the two databases is provided in the toolkit, and instructions on its use can be found in the back pages of the Data Entry Guidelines. Data Analysis Once survey data have been entered into a LIAT database using a data entry tool, and the data have been checked for missing elements or entry errors, the survey coordinator or data analyst should begin to produce preliminary results for debriefing in-country stakeholders. Data analysis can be challenging due to the limited time that is typically allowed for calculating the data and organizing results in an easy-to-understand format. Standard Indicators for LIAT Analysis Analysis of data collected with a LIAT will generally follow the associated LIAT indicators summarized in Table 4 and provided in complete detail in the document, Monitoring and Evaluation Indicators for Assessing Logistics Systems Performance, available as part of this toolkit. These indicators are designed to address the logistics issues that are of most interest to stakeholders. They can also be used as part of the assessment planning process by developing an analysis plan and showing interested parties what information will be available to them through the LIAT. Although the standard LIAT indicators are comprehensive and are recommended for presenting preliminary data, they are not the only indicators that can be used to analyze the data collected in a standard assessment using a LIAT. Additional indicators can be developed to fit the particular needs of a product area or logistics program, for example, determining the percentage of facilities where different health commodities are delivered together. Table 4 shows a list of indicators taken from a recent assessment in which the LIAT was used. These indicators are the most common ones used for measuring the status of a logistics system and the quality of the source from which data will be collected. Although the list of indicators is introduced in this section, indicators are also an important part of the assessment planning process. Discussing the list of indicators with stakeholders before the assessment begins will inform everyone concerned about the potential resources needed for data collection (i.e., the time and staff it takes to collect the data and how that equates to costs) and what results they can expect from the activity. 40 Table 4. List of Sample Indicators for Evaluating a Logistics System Indicators Data Source(s) Training Percentage of personnel who learned logistics through formal training Respondent Percentage of facilities with ledger books available and updated by product Ledger books Percentage of facilities with accurate stock balances in ledger books Ledger books and physical inventory count Percentage of facilities using LMIS forms for reporting and ordering Respondent Percentage of facilities regularly submitting LMIS forms Respondent Logistics Management Information System Inventory Control Percentage of facilities that ordered according to maximum stock levels Average lead time for ordering all commodities Transportation Percentage of facilities where commodities are collected by facility staff Supervision Percentage of facilities that report receiving logistics supervision visits Storage Percentage of facilities that maintain acceptable storage conditions Stock Status Respondent Respondent Respondent Respondent Visual observation Percentage of facilities experiencing a stockout of commodities on the day of visit Ledger books and physical inventory Percentage of facilities experiencing a stockout of commodities in the previous six months Ledger books Average duration of stockouts for commodities in the previous six months Ledger books Average months of stock on hand for all products Ledger books and physical inventory Standard LIAT Analysis Process Data analysis begins immediately following the entry and “cleaning” of all completed surveys; analysis should not be done when data from surveys still need to be added to the database. Questions or issues with surveys will need to be resolved before results can be shared with stakeholders. The time it may take a data analyst to process data greatly depends on the software available for data analysis and when the results are needed. Statistical software packages, such as SPSS, SAS, and 41 STATA, are designed to allow data analysts to deal with large amounts of data in a relatively short time, as long as the operator is familiar with the software. Other software, such as Microsoft Access or Excel, can be used for data analysis, but dealing with data collected from large surveys may be cumbersome using these relatively simple tools and, without preprogrammed cells for calculations, the analysis process may take some time to complete. Knowing what software analysis tools will be used should reveal how many days will be needed for analyzing the data. A typical scenario where the data analyst is familiar with SPSS can produce preliminary results within two to three days. For SPSS users, a standard analysis document is available in Appendix 12. This document explains how the indicators are calculated and suggests SPSS syntax to use in the analysis. Box 6. Chapter Summary: Using the LIAT  The Logistics Indicators Assessment Tool (LIAT) is a quantitative data collection instrument used to conduct surveys that assess health commodity logistics system performance and commodity availability at health facilities.  Using the LIAT to conduct an assessment based on experimental survey design (i.e., data collection from an intervention group and a control group) will produce the most robust results, but using a more cost-effective non-experimental design (i.e., data collection from an intervention group only) can still produce extremely valuable and accurate data for decisionmaking in supply chain management.  Because it is rarely possible to visit all facilities serviced by a supply chain, a representative sample should be chosen for data collection. Based on feasibility, survey organizers should agree on how many facilities to randomly select at each level of the system and across what geographic area in order to support their efforts to make evidence-based recommendations about the entire system.  An assessment team is made up of data collectors (including leaders for each data collection team), a survey coordinator, and data entry staff. Optional personnel include survey monitors and data analysts. Trainers should be engaged to train all team members appropriately in the specific data collection needs of the survey, regardless of the participants’ previous experience.  Data collection at a primary health facility using the LIAT usually takes between two and four hours. Teams should be able to visit at least two primary health facilities each day. To better ensure accessibility, facilities should be informed ahead of time about the visit, and, before conducting a survey in a new administrative area, the team leader should bring the entire team for a visit with the local government office to brief officials on the assessment activity.  Controlling the quality of data collected is an important daily activity. Data collectors should review their work before leaving a facility, and team leaders should facilitate a daily review of all completed LIAT forms to ensure clarity and completeness. Collecting geographic coordinates using global positioning software devices can facilitate a better understanding of geographic trends that affect areas served by a supply chain.  Data may be submitted for entry on a rolling basis (using the services of a courier) or upon return from field visits. Use of electronic data collection devices, such as PDAs, can eliminate the need for data entry staff and reduce the amount of data “cleaning” that must be undertaken to avoid errors.  Data analysis is best conducted using an establish software package. Standard LIAT indicators allow for analysis of the logistics issues that are of most interest to stakeholders, including training, logistics management information systems (LMIS), inventory control, transportation, supervision, storage, and stock status. See Appendix 13 for a summary of preparatory steps that assessment organizers should take before beginning training, data collection, and data entry. 42 Presenting Results Assembling Key Information In most assessments, preliminary results are presented to in-country stakeholders directly following the completion of data analysis, and a final technical report is issued shortly thereafter. When the LIAT and LSAT are used jointly in an assessment, information from both sources should be presented to best depict the situation on the ground. If the LIAT is used independently, it is useful to gather and provide additional information as background on how the logistics system being assessed is designed. Similarly, assessments that use only the LSAT should be complemented with supplemental data on supply chain efficiency, whether gathered from recent studies or a limited set of site visits. Presenting the Results The presentation of results should respond to all of the objectives and concerns that stakeholders expressed at the outset of assessment planning. Although it is important to provide evidence in easy­ to-understand language, presenters should be careful to avoid simplifying their information to the point of being less useful or even condescending. Ideally, reports should be available in local languages for non-English-speaking stakeholders. Visual representations in presentations and technical reports should also be easy to understand. Whereas data-rich, complex graphs and charts might be appropriate for fellow evaluators, counterparts are more likely to discuss, share, and act upon results that are provided through clearly labeled graphics, data tables, and with other media (e.g., photos, maps, etc.). Presenting the results of an assessment should be done well in advance of their intended use. Far too often, assessments are conducted immediately before a strategic planning session or drafting a program’s work plan. Assessment planners should budget ample time for presenting, writing, and disseminating results so that stakeholders have an opportunity to respond to the findings before using them for planning and decisionmaking. Last, authors of in-country presentations and technical reports must be mindful of the potential consequences that assessment results entail. For example, despite efforts to ensure the confidentiality of data, some individuals may be blamed for specific logistics weaknesses that were detected. Or an outside service provider (i.e., a private company contracted to transport commodities) may strongly dispute results that it feels could reduce its likelihood of obtaining future contract opportunities. In cases where a system weakness is directly attributable to a specific unit or organization, the findings should be shared privately in advance with that organization and the manager or supervisor in charge. Any conclusions drawn should be validated during the assessment and presented with supporting data and documentation. 43 Mapping Results Simple maps are a particularly effective way to present data trends that are identified during LIAT analysis. For example, using a map to show facilities where frequent commodity stockouts occurred and those that experienced infrequent or zero stockouts can clearly indicate regional effects and provide a focal point for discussion between stakeholders. Maps of this type are also easier to understand for decision makers and other important audiences whose members may be less familiar with logistics principles. However, concerns of facility confidentiality must be considered when presenting results in this manner. A number of different Geographic Information Systems=(GIS) software packages are available for map making. In particular, there are open source GIS software options that are simple to use and free-of-cost. DIVA-GIS (available at www.divagis.org) is one example of software that can be used to display LIAT data visually and conduct some initial spatial data analysis. For more advanced spatial analysis, a tool such as tQGIS is useful. Storing Data and Results for Follow-On Activities Documentation from the assessments should be carefully retained to enable follow-on assessment and, ultimately, an understanding of supply chain changes over time. While it is recommended that all documents supporting the final assessment report should be saved, particular attention should be paid to the finalized database (often stored in SPSS), LSAT and LIAT questionnaires, technical report, and facility list (including replacement facilities). With these three documents, a follow-on assessment team will be able to visit the same sites and compare newly obtained data to previous information. The stakeholder that commissioned the activity should receive all three of these documents. However, similar to the issue of confidentiality surrounding sharing preliminary results, consideration should always be given to ensuring the continued confidentiality of interviewees and facilities from which data were collected. Box 7. Chapter Summary: Presenting Results  When the LIAT and LSAT are used jointly in an assessment, information from both sources should be presented to best depict the situation on the ground. If the LIAT is used independently, it is useful to gather and provide additional information as background on how the logistics system being assessed is designed. Similarly, assessments that use only the LSAT should be complemented with supplemental data on supply chain efficiency, whether gathered from recent studies or a limited set of site visits.  Assessment planners should budget ample time for presenting, writing, and disseminating results so that stakeholders have an opportunity to respond to the findings before using them for planning and decision making.  The presentation of results should respond to all of the objectives and concerns that stakeholders expressed at the outset of assessment planning. However, presenters should be mindful of the sensitivity of sharing preliminary results, particularly in the case of information that may compromise the confidentially of data collection sites, or that have potential repercussions for staff members or entire organizations.  Simple maps and other graphic representations are key ways of increasing stakeholders’ understanding of technical issues and facilitating the sharing of results with other concerned parties.  Documentation from the assessments should be carefully retained to enable follow-on assessment and, ultimately, an understanding of supply chain changes over time. 44 For more information, please visit deliver.jsi.com. USAID | DELIVER PROJECT John Snow, Inc. 1616 Fort Myer Drive, 11th Floor Arlington, VA 22209 USA Phone: 703-528-7474 Fax: 703-528-7480 Email: askdeliver@jsi.com Internet: deliver.jsi.com

View the publication