Handbook for family planning operations research design.

Publication date: 1998

Handbook for Family Planning Operations Research Design SecondEdition AndrewA. Fisher JohnE.taing JohnE.Stoe_el JohnW.Townsend Population Council One Dag Hammarskjold Plaza New York, New York 10017 www.popcouncil.org Library of Congress Andrew A. Fisher, Sc.D. is Senior Associate Cataloging In Publication Data and Director of the Africa Operations Be- search and Technical Assistance Project (Afri- Handbook for family plannin 8 operations research ca oRfrA Project), The Population Council, design / Andrew A. Fisher. let al.] -- 2nd ed. Nairobi, Kenya. p. cm. l_ev. ed. of: Handbook for family planning operations research desisn / Andrew Fisher,' John E. Laing, Ph.D. is Consultant, Austin, John Lalng, John Stoeckel. Texas. He was formerly a Senior Associate Includes bibliographical references, with The Population Council in South and ISBN 0-87834-059-9 East Asia. 1. Birth control--Research--Handbooks, manuals, etc. John E. 8toeckel, Ph.D. is Senior Associate, 2. Operations research--Handbooks, manuals, The Population Council, Bangkok, Thailand. etc. I. Fisher, Andrew, 1941- II. Fisher, Andrew, 1941-Handbook for family plannir_ JOhn W. Townsendp Ph.D. is Senior Associ- operations research design, ate and Senior Representative for Latin Ameri- HQ763.5.H36 1991 ca and the Caribbean, The Population Coun- 363.9'6'072073--clc20 91-6284 CIP cil, Mexico City, Mexico. Any part of this Handbook may be copied or adapted to meet local needs without permission from the authors or the Populatio n Council, provided that the parts copied are distributed free or at cost (not for profit). Any commercial reproduction requires prior permission from the Population Council. The authors would appreciate receiving a copy of any materials in which the text from the Handbook has been used. Printed in the United States of America First edition 1983 Secondedition 1991 Second printing 1998 Contents. Preface to the Second Edition vii 3 A Special Population Group is Acknowledgments ix Affected 4 It Relates to Ongoing Program Activities 5 There are Broad Social, Economic, Health, or PoliticalImplications 6 It is Considered Important by 1 Introduction 1 Many Different Groups A What Is Operations Research? 1 B The Focus and Objectives of Operations Research 1 ¢_ C Categories of Operations 0¢ Research Studies 1 1 Exploratory/Diagnostic Studies 3 Strategy Selection to Solve the 2 Field Intervention Studies Problem 14 3 Evaluative Studies A Using a Quality-of-Care D The Methods and Study Framework 14 Designs of Operations Research 2 B Examples of Strategies to Test 14 C Guidelines for Selecting an Appropriate Strategy to Test 15 2 AI 2 Identifying, Defining_ and /_ Justifying the Research Problem 4 4 Objectives and Hypotheses 18A Problem Identification 4 A Ultimate Objectives 18B Problem Definition 7 1 Incidence and Prevalence B Immediate Objectives--Who 2 Geographic Areas Affected will do, how much of what, to 3 Characteristics of Population whom, when, where, and for Groups what purpose 19 4 Probable Reasons for the C Hypotheses 19 Problem 1 Relationships among Variables 5 Possible Solutions a Independent Variables 6 Unanswered Questions b Intervening Variables C Problem Justification 11 c Dependent Variables 1 It is Current and Timely 2 Direction of Relationship 2 A Large Population is Affected 3 Conditions of Relationship JH Ill E Guidelines for a "Good" Research Design 38F The Principle of the Three Multiples 38 5 Intervention Description 23 1 Multiple Data Sources A Who Will Be Responsible? 23 2 Multiple Measurements Over B Where Will Activities Take Time Place? 23 3 Multiple Replications C What Activities Will Be Initiated? 23 8 8 Sampling 4o A Selecting Area and Population 40 6 Operational Definitions 26 B Selecting Cases 40 A Operational Definitions of 1 Probability Sample Variables 26 a Simple Random Sampling 1 Indicators b Systematic Sampling 2 Variable Categories c Stratified Sampling a Mutually Exclusive d Cluster Sampling b Totally Inclusive e Multistage Sampling B Operational Definitions of f Probability Proportional to Terms 27 Size (PPS) 2 Non-probability Sample C Sample Size Determination 43 Study Design 30 q7 A Reliability and Validity 30 B Validity Threats 30 9 Data Collection 47 1 History 2 Selection A Quantitative Data 47 3 Testing 1 Structured Interviews 4 Instrumentation 2 Service Statistics 5 Maturation 3 Self-Administered Questionnaires 6 Mortality 4 Secondary Data Sources C Types of Study Designs 32 B Qualitative Data 501 Unstructured Interviews 1 Nonexperimental Designs 2 Focus Group Discussions a Posttest-Only Design b Pretest-Posttest Design 3 Direct Observation of Operations 4 Content Analysis of Written c Static-Group Comparison Materials 2 Experimental Designs a, Pretest-Posttest Control Group C Data Quality Checks 51 Design D Confidentiality of Information 52 b Posttest-Only Control Group Design 10 3 Quasi-Experimental Designs a Time Series Design b Nonequivalent Control Group Design 10 Tabulation and Analysis of Data 53 c Separate Sample Pretest- A Preparing Tabulations 53 Posttest Design 1 Coding Data D Selecting a Study Design 37 2 Editing Coded Data 1 Ethical Issues B Plan for Data Analysis 54 2 Practical and Administrative 1 Attributes of the Data Issues a Central Tendency of the Data 3 Technical Issues b Variance in the Data iv c Differences within the Data d Relationships within the Data 132 Types of Analytical Procedures a Variable Transformations b Univariate Analysis 13 Limitations of the Study 68 Nominal Measurement A Design and Analysis Ordinal Measurement Limitations 68 Interval Measurement B Special Situational Factors 68 c Time Series Analysis d Comparisons e Bivariate Relationships Between Nominal Variables Between Ordinal Variables 1 4Between Interval Variables f Multivariate Analysis All Interval Independent 14 Resources and Facilities 69 Variables A Available Resources and Both Interval and Categoric Facilities 69 Independent Variables B Study Budget 69 All Categoric Independent Variables g Cost-Effectiveness Analysis h Use-Effectiveness Analysis 15i Fertility Analysis C Dummy Tables 63 15 Appendixes 7111 11 Dissemination of ResearchFindings 64 16 A Dissemination Strategy 64 B Writing Reports 64 16 Title Page and Abstract 72 12 12 Utilization of Study Findings 66 Bibliography 75 Prefaceto the SecondEdition The Handbook was first published in English by the Population Council in 1983 and was based on the authors' experience with field research studies in Asia. Subsequently, it has been translated into French, Spanish, Thai, Bahasa Indonesia, Arabic, and Chinese. It has been widely used not only in Asia but also in Latin America and Africa. In this second edition of the Handbook, we have maintained the same basic format, but we have revised and expanded some of the sections. Where appropriate, examples from Latin America and Afiica have been added. The introductol?¢ section contains a more complete and current statement on the process of health and family planning operations research. New chapters have been included on selecting an appropriate intervention to test in an OR study, and on describing the main elements of the study intervention. The chapter on information dissem- ination has been expanded and a new chapter on the utilization of research findings has been added. The Handbook assumes some familiarity with the terms and concepts of research design and statistics, as well as some experience with field research studies. It has several objectives. First and foremost, it is specifically designed to help health and family planning researchers .develop and write a detailed operations research proposal. It has often been used in workshops as a guide for preparing a well-written and detailed research study proposal. It is also intended to help program administrators and managers understand the process of operations l_esearch and the uses of research findings for seivice delivel 3, improvement. Finally, although the Handbook is not a text on research methods, it does provide a review of key concepts and important methods essential to conducting field researeh studies. vii Acknowledgments The Handbook was originally written as training matelial incidental to the US Agency for Interna- tional Development (USAID), office of Population Contract No. DPE-0632-00-1029-00, Family Planning Operations Research in Asia. This revised edition was written as part of the USAID Office of Population Contract DPE-3030-Z-00-8065-00, Strategies fur Improving Family Planning Service Delivery, The Africa Operations Research and Technical Assistance Project {Afi'ica OR/TA Project) and Contract No. DPE-3030-Z-00-9010-00, Operations Research Project to Improve Family Plan- ning in Latin America and the Caribbean, Operations Research and Technical Assistance in Latin America and the Caribbean (INOPAL). The authors are particularly grateful for the constructive comments received fi_om many workshop participants and other users of the Handbook in Asia, Afi'ica, and Latin America. We are also thankful for the comments, support, and encouragement of colleagues at USAID, Cooperating Agencies, Columbia University, the University of Michigan, Johns Hopkins University, Tulane University, and the University of Hawaii. As fax" as possible, and without radically altering the original format, we have tried to incorporate these comments into this t_evised edition. Finally, We greatly appreciate the critical reviews and many helpful suggestions provided by out" l]'iends within the Population Council: lan Askew, Beverly Ben Salem, John Bongaarts, George Brown, Maria Elena Casanova, Herve Ludovic de Lys, Mamadou Diallo, James Forfeit, Joanne Gleason, Dale Huntington, Anrudh Jain, Michael Koenig, Federico Leon, Toni Martin, Peg_ McEvoy, Robert Miller, Cecilia Ndeti, Lewis Ndhlovu, Jose Galicia Nunez, Sandra Rosenhouse, Pauline Russell-Brown, Diouratie Sanogo, Valeda Slade, Placide Tapsoba, Luis Varela, and Ricardo Vernon. ix Introduction A What Is Operations Research? National health and family planning programs in Asia, Africa, and Latin America are intended to have an impact on birth, death, and morbidity rates. These programs are large in expenditure, personnel employed, geographic coverage, and component activities. They are complex under- takings that require detailed planning at all levels, close coordination of program effort, careful training and supemdsion of personnel, and continuous evaluation of program development and impact. Operations research (OR) is a primary means available to support these essential plan- ning, coordinating, training, and evaluation functions. OR is a process, a way of identifying and solving program problems. As currently applied in health, family planning and other development programs, operations research can be defined as a continuous process with five basic steps: 1) problem identification and diagnosis, 2) strategy selection, 3) strategy experimentation and evaluation, 4) information dissemination, and 5) infor- mati_on utilization. The process of OR is designed to increase the efficiency, effectiveness, and quality of services delivered by providers; and the availability, accessibility, and acceptability of services desired by users. B The Focus and Objectives of Operations Research Operations research focuses on the day-to-day activities or "operations" of health and family planning programs. These operations are under the control of managers and administrators, such as training, logistics, information and education activities, clinic activities, and rural service delivery systems. OR looks at problems affecting the supply side of programs, not the demand side. Indeed, OR is characterized by its focus on current supply or service delivery problems and its search for solutions, or in the language of research, variables that can be manipulated through administrative action. Operational health and family planning research yields answers to perceived program problems. An important objective of OR is to provide managers, administrators, and policymakers with the information they need to improve existing delivery activities and plan future ones. OR seeks practical solutions to .problem situations and viable alternatives to unsatisfactory operating methods. It diagnoses and evaluates the problems of programs and compares one service delivery approach against another in terms of impact, cost effectiveness, quality, and client acceptability. C Categories of Operations ResearchStudies Operations research studies can be classified under three headings: 1 Exploralory/Diagnostie Studies. These studies seek to determine the parameters of a problem situation affecting service delivery. They examine the underlying supply factors influencing the effectiveness, efficiency, and quality of program ser4ices. Exploratory/ diagnostic studies are retrospective or cross-sectional in design. This type of study is needed whenever there is a perceived program problem but the nature of the problem 1 simply is not known. A key aspect of these studies is the search for programmatically manipulable variables. 2 Field Intervention Studies. These studies test, on an experimental basis, new ap- proaches to overcoming a program problem. In many situations, the factors responsible fox- a program problem are known but the most efficient and cost-effective means of alleviating the problem are not known. Field intelvention studies test new service delivery modes. These studies are always prospective and longitudinal and usually employ either an experimental or quasi-experimental research design. 3 Evaluative Studies. Very often health and family planning activities are implemented for years but never assessed. In such cases, evaluative studies can be a valuable opera- tions research approach for examining retrospectively or cross-sectionally the effect of program activities. These three categories of OR studies are not mutually exclusive. Frequently a single OR study will begin with an exploratory/diagnostic phase to iden'tify key variables of importance. During the second phase, a field intervention might be initiated and then followed by an evaluative phase to determine the impact of the intervention. D The Methods and Study Designs of Operations Research The methods of OR range from the qualitative to the quantitative, the study designs from the nonexperimental to the true experimental. There is no single set of methods or designs unique to operations research. Indeed, it is not the application of a particular set of methods or the use of one design over another that distinguishes OR from other forms of research. Rather, it is the focus or objective of this type of research. Simply stated and in its broadest terms, the objective of operations research is to improve the delivery of services. While OR studies may employ experi- mental or nonexperimental designs, and they may include a quantitative analysis of demographic processes or a qualitative discussion of health issues, the central objective always is to obtain a better understanding of the "operations" of programs so that needed improvements can be made. Illustrative Topics for OR Studies Hundreds of health and family planning OR studies have been completed in Asia, Africa, and Latin America. For the purpose of illustration only, we list below nine general areas that often are the focus of operations research studies. 1 Training Programs. Countless health and family planning workers have received training • throughout the world. Frequently, OR studies are conducted to examine the content and methods of training, or to compare one type of training approach against another in terms of field-worker knowledge and performance. 2 Information, Education, and Communication (IEC). Providing information to people is a major activity of most health and family planning programs. OR studies are often designed to compare different communication approaches in terms of message understanding, message retention, cost-effectiveness, and the ability to reach specific target audiences. 3 Management Information Systems. All health and family planning programs collect infor- mation on clients and services delivered. Sometimes, however, the information is inaccurate or simply not used for any administrative purposes. OR studies can be designed to experiment with new data collection systems and to test new procedures for using the information collected to improve services. 4 Program Impact. OR studies frequently test prospectively and in field settings the impact of different approaches to service delivery, such as the use of community-based distributors (CBDs) or traditional birth attendants (TBAs). Impact can be measured in many different ways by examining, for example, contraceptive prevalence, contraceptive continuation rates, client satisfaction with services, or increased availability and accessibility of sewices. 5 Administration and Management. Qualitative assessments of program administration and management are sometimes the focus of OR diagnostic studies. 6 Quality of Care. Increasingly OR studies have been directed at evaluating the quality and acceptability of services offered to clients. These studies can provide an analysis of the current situation of health and family planning services. 7 Commercial Channels of Distribution. Many family planning programs use commercial outlets such as pharmacies to distribute nonprescription contraceptives. OR studies can be designed to compare commercial against noncommercial service delivery systems. 8 AIDS (HIV) and Family Planning. In many countries program administrators are concerned about the potential impact AIDS (HIV) may have on family planning activities. In Africa and Latin.America, a number of OR studies have been conducted to test the effect of integrating AIDS (HIV) information and education into on-going family planning programs. 9 Male Involvement in Family Planning. In some national family planning programs, attention has been directed almost exclusively at females and most services are available only at MCH centers. Testing mechanisms for involving males in family planning programs and providing them with appropriate services either at the work place or elsewhere has been a focus of many OR studies. Obviously there are many other topics that could become the focus of an operations research study. Whatever the topic selected, it is important to remember that the primary goal of operations research is not merely to collect information and discover the reason for a problem situation. It is to use the information collected to solve the problem. Designing and conducting an OR study is only one part of the entire operations research process. Disseminating the results from the study and then utilizing these results to improve service delivery constitutes the other essential part of the OR process. Identifying, Defining, and Justifyingthe ResearchProblem A Problem Identification There are countless problems with family planning and health programs. Finding a problem is not hard, but identifying one for the purpose of research is not always easy. One of the most important tasks of research is to identify and define clearly the problem you wish to study. If you are uncertain about the research problem, if you are not certain in your own mind about what you want to study, you may be sure that others who read your proposal will also be uncertain. A well-defined problem leads naturally to the statement of research objectives, to the hypotheses, to a definition of key variables, and to a selection of a methodology for measuring the variables. A poorly defined research problem leads to confusion. All research is set in motion by the existence of a problem. A problem is a perceived difficulty, a feeling of discomfort with the way things are, a discrepancy between what someone believes should be and what is. While problems are the initiating force behind research, not all problems require research. A potential research situation arises when three conditions exist: 1 A perceived discrepancy between what is and what should be. 2 A question about why the discrepancy exists. 3 At least two possible and plausible answers to the question. The last point is important. If there is only one possible and plausible answer to the question about the discrepancy, then a research situation does not exist. Consider the example given below. 1 Example of a Nonresearch Problem Problem Situation: A recent survey in District A found that 1,000 women were continu- ous users of contraceptive pills. But last month's service statistics indicate that none of these women are using contraceptive pills. Discrepancy: All 1,000 women should be using contraceptive pills, but all 1,000 women are not using contraceptive pills. Problem Question: What factor or factors are responsible for 1,000 women discontinu- ing their use of contraceptive pills? Answer: A monsoon flood has prevented all new supplies of pills reaching District A, and all old supplies have been exhausted. In the above example, a problem situation exists, but the reason for the problem is already known. Therefore, assuming that all the facts are correct, there is no reason to conduct research on the factors associated with pill discontinuation among the 1,000 women. On the other hand, there may very well be a need to conduct research on the question of why the supply and logistics system is incapable of providing contraceptives to women during the monsoon. Study the next example. 4 2 Example of a Research Problem Problem Situation: District A is always flooded during the monsoon season. Recogniz- ing this problem, the national family planning program estab- lished a new supply logistics system for the district. Each pill user is given a four-month supply before the monsoon begins. During the monsoon, small motorboats are available to transport new supplies to selected distribution centers accessible to village-level family planning workers. Despite these new measures, this year service statistics indicated that there are no pill supplies in District A. Discrepancy: The new logistics system should be able to assure a continuous supply of pills, but this year there are no supplies. Problem Question: Why has the new supply logistics system been incapable of deliv- ering contraceptive pills to users? Possible Answers: 1 An order for new pill supplies was not placed in time before the monsoon rains. 2 The riverboats used to transport the supplies are out of order. 3 Field-workers were not told about the new system and failed to give users a four-month supply of pills before the monsoon. In this example, there are several possible and plausible reasons for the problem situation. One or more of these reasons might be correct. Therefore, this is a potential research situation. In some situations, it is relatively easy to identify the problem, to define it, to hypothesize the reasons for it, and to conduct research to determine which reason is correct or more nearly correct. The reasons for the supply and logistics problem in the above example could probably be determined fairly easily and certainly would not require an expensive research study. Other problems, such as the one in the next example, are not so easy to identify or to study. 3 Example of a Research Problem Problem Situation: A recent family planning survey revealed great differences between villages in the rate of contraceptive prevalence. Despite the fact that all villages receive the same level of health and family planning services, some villages have a prevalence rate as high as 80 percent, while others have a rate as low as 6 percent. Discrepancy: All villages should have approximately the same rate of contra- ceptive prevalence, but in fact there is great variation between villages. Problem Question: What factors are responsible for the areal variation in contraceptive prevalence rates? Possible Answers: 1 Villages differ in their socioeconomic environments. Some vil- lages are agricultural; some are fishing communities. Some vil- lages are Hindu; others are predominantly Muslim or Buddhist. Some villages have access to markets in towns; others do not. Some villages have schools, health clinics, electricity, and a good water supply; others do not have these facilities. These socio- economic differences affect the level of contraceptive practice. 2 Villages differ in institutional support for contraceptive accep- tance. In some villages, local influentials strongly support the national family planning program. In other villages, they do not support it. In some villages, there are active Mothers' Clubs that support family planning. In other villages, there are no Mothers' Clubs. These differences in institutional support for family plan-. ning affect the level of contraceptive practice. 5 3 Village-level health and family planning workers differ in their effectiveness. Some workers are highly motivated and very active ill their assigned areas. Other workers are less motivated and less active. These differences in worker effectiveness affect the level of contraceptive practice. While the problem situation presented ahove is fairly clear, the reasons for tile problem are complex. Three reasons have been given, but it is likely many more could be stated. In situations such as this, the researcher must devote considerable time and attention to identi- fying the problem situation. The aim is to focus the research on the most important aspects of the problem. 4 Example of a Research Problem Problem Situation: During their periodic visits to clients, the supervisors in a rural, community-based family planning program observe that many users of oral contraceptives are generally poorly informed about the appropriate use of the pill. For example, 58 percent of the women consulted had taken the pill incorrectly during the past month. Some had waited either too long or not long enough after the menstrual cycle; others did not know what to do when they missed taking a pill. Discrepancy: The family planning program prides itself on providing high-quality services. Nearly all of the rural health promoters who provide infor- mation and supplies to users received three days of training upon entry into the program. They also receive supervision when they are resupplied with methods. All users should know how to correctly use their method of choice, but clearly knowledge is lacking. Problem Question: How can program management guarantee an acceptable level of quality of use in its rural community program? Possible Answers: 1 The program has potentially faulty selection procedures for can- didates for the voluntary position of rural promoters. The pro- cedures need to be reviewed and possibly changed. 2 The traditional three-day training program is inadequate in terms of content and practice of communication with clients. The train- ing program should be revised. :3 The supervision system gives equal time to each promoter re- gardless of the needs of the community or the skills of the promoter. The process of supervision should be examined. 4 The management of the program is not really concerned about quality in the community program in that it accepts any level of competence among promoters. Quality-of-care elements should be incorporated into the program. _ The range of services the promoter is required to provide is too large and complex for most voluntary personnel. The range of services should be changed. In this example the problem situation is clear. Family planning supervisors have observed that many oral contraceptive users are poorly informed about the use of the pill. A discrepancy exists. Pill users should know how to take the pill correctly, but most don't know how to take the pill correctly. The discrepancy between what should be and what is suggests a problem question and five possible answers to the question. It is not known which of these five possible answers are correct or morenearly correct. One important way that OR accomplishes the first step of problem identification is to involve not only the researchers but also administrators (and others, as appropriate, such as village chiefs, teachers, and local leaders) in the entire process. This involvement combines the program experience and problem understanding of administrators and local leaders with the technical, methodological skills of researchers. Teaming researchers and program administrators is an educational experience for both that can have long-range outcomes which go far beyond the mere design and implementation of a single OR study. Researchers, for example, begin to understand more fully the day-to-day concerns of administrators and the problems faced by seivice delivery organizations. This increased understanding can help to sharpen the locus of a study on those aspects of a program problem that are amenable to change. Administrators, on the other hand, begin to appreciate the need to identify and define program problems on the basis of accurate data. They begin to view research as an important tool for decision-making and as an on-going process to which they can contribute. Their early involvement in the operations research process is more likely to increase their interest later in reviewing and utilizing the results from OR studies. B Problem Definition Identifying a problem situation is the first essential step in designing a research proposal, but it must then be followed by a process of problem definition. The research problem identified must now be defined in terms of its occurrence, intensity, distribution, and other measures for which data are already available. The-aim is to determine all that is currently known about the problem and why it exists. While it is always possible to guess why a problem exists, guesses are often wrong and usually they do not provide a firm basis for designing a research study. A far better way to define a problem situation is to review relevant literature, examine current selvice statistics, seek educated opinions from pei_ons concerned with the problem, and obtain probable reasons for the problem from social, economic, or health theoi_y. A careful review of existing sources of information on a problem and an epidemiologic diagnosis help the investigator determine: 1 Incidence and Prevalence. An epidemiologic diagnosis should always be made of problems related to health and family planning. In other words, how widespread is the problem? What is its distribution? How often does it occur? An epidemiologic diagnosis will help establish the parameters of the problem. 2 Geographic Areas Affected. It is important to know if there are particular geographic areas affected by the problem. Does the problem generally occur in rural areas only? Does it also affect cities? Is the plx)blem l_estricted to mountain areas, coastal areas, or island areas? 3 Characteristics of Population Groups. Are there special population groups affected by the problem, such as mothers, children, teenagers, newly married couples, or women over age 35? 4 Probable Reasons for the Problem. A review of information on a problem should suggest a number of probable reasons why the problem exists. What is the current thinking about the reasons for the problem? Is there general agreement among many people about the reasons, or are there many different, conflicting views? 5 Possible Solutions. Many projects and programs may have been directed at the problem in an attempt to overcome it. What types of solutions have been tried in the past? How successful have past efforts been? What approaches to solving the problem seem to work? What approaches seem not to work? 6 Unanswered Questions. From the review of information on the problem, what seem to be the unanswered questions about the problem? What aspects of the problem need to be researched further? Reviewing what is already know about a problem situation is an essential part of the research process. A good review of information will suggest the social, economic, political, and health importance of the problem. It will help to narrow the focus of the proposed research. It will indicate the major theoretical concepts and operational variables other researchers have consid- ered important. It will suggest possible research hypotheses that need to be tested. Finally, it will help the investigator avoid reinventing the Wheel, or in other words, conducting research on a problem that has already been researched countless times in the past with fairly consistent findings from each study. I Example of a ResearchProblem Identification and Definition Statement: Asia 1 Although substantial progress toward reducing fertility rates was made during the last decade in several Southeast Asian countries, significant program constraints still exist and threaten further fertility declines. In Indonesia, the relatively high contraceptive prevalence rate on Java and Bali of 42 percent must be viewed against the relatively low rate of 17 percent in the Outer Island I area and only 6 percent in the Outer Island II area. in the Philippines, only 25 percent of all contraceptive practice involves the use of modem methods. In Malaysia, new sterilization acceptors constitute less than 6 percent of all new acceptors and new IUD acceptors less than 3 percent. In Thailand, the vasectomy program is lagging despite some remarkable acceptanqe rates demonstrated when private sector teams have taken services directly to villages. 2 Most program administrators within Asia are aware that a new level of effort and direction will be needed in order to sustain the trends established in the 1970s. In Indonesia, attention needs to be given not only to the Outer Island areas but also to the urban areas, particularly Jakarta. Although Jakarta has a health and family planning service delivery system designed to cover all areas of the city, the contraceptive preva- lence rate is only 21 percent of currently married women, or in other words, approx- imately half of the rate for all of Java and Bali. Other large urban areas in the country also lag behind the rural areas. 3 While the factors responsible for the difference in contraceptive prevalence rates be- tween urban and rural areas in Indonesia are not known, it has been suggested that urban areas lack the traditional structures of community organization and leadership that help support the family planning program in rural areas. Another possible reason for the difference might be lack of awareness, particularly among new migrants to urban areas, of sources of contraceptive supply. Many of the new migrants have low levels of education, do not have access to sources of information, and are unaware of the location of health and family planning service centers. Also, family planning city workers often find it difficult to provide continuous services to new migrants at their homes since these people tend to shift their residence periodically from one area of the city to another. It is likely that, among acceptors, these shifts result in discontinuation, or at least in an interruption of contraceptive practice. 4 In the past, several attempts have been made to increase the urban rate of contraceptive prevalence by providing more accessible services to residents. In Jakarta, mobile 'vans have been used to bring services to outlying areas of the city. Another approach has been to distribute condoms and pills through such commercial outlets as small shops and tea stalls. While both of these approaches are useful; .'they also have significant drawbacks. Mobile vans are expensive to operate, and they Can reach only a relatively small proportion of the population of potential acceptors. Commercial outlets are helpful for current acceptors, but they are not very effective as a means to motivate new acceptors. Also, these outlets cannot provide needed maternal and child health services, such as immunizations and pre- and postnatal care. 5 What is needed in urban areas is some means of (1) reaching large numbers of current and potential acceptors and (2) providing them with a range of health and family planning services. An important unanswered question for the national family planning program is how this can best be accomplished. 6 One approach that has been suggested but so far not tried is to train a special category of urban health and family planning personnel who would contact acceptors and potential acceptors at their places of employment. With the assistance of employers and unions, the urban health and family planning worker could establish service delivery centers at factories, government offices, hotels, construction sites, markets, and other places of employment. It is likely that this approach would be less costly than the use of mobile vans; it would be capable of reaching large numbers of people; and the range of services that could be provided would be greater than that provided by commercial outlets. 8 2 Comments on the Example In the first paragraph of this example, a broad problem situation is identified. Substantial progress toward reducing fertility rates in several Southeast Asian countries has been made, but continued progress is threatened by significant program constraints. Sevei'al examples of program con- straints are then given. These examples serve to indicate the geographic areas affected by the problem. In the second paragraph, the problem of program constraints is narrowed down to just one country--Indonesia. Also, attention is focused on just one program problem--the difference in contraceptive prevalence rates between urban and rural areas. A discrepancy is indicated. Jakarta has a health and family planning service delivery system designed to cover the entire city. This system should result in prevalence rates equal to the rate in rural areas. But the prevalence rate in Jakarta is only half the rate found in all of Java and Bali. By the end of the second paragraph, we know the specific problem situation--urban/rural differences in contraceptive practice. We know the location of the problem--Indonesia, urban areas, particularly Jakarta. We know the magnitude of the problem--21 percent prevalence in Jakarta compared with 42 for all of Java and Bali. We know the specific population of interest--urban residents. In the third paragraph, several possible reasons for the problem situation are suggested. It is not known which one of these reasons is correct or more nearly correct. .The fourth paragraph outlines several possible solutions to the problem of low prevalence rates in urban areas. Mobile vans have been used. Commercial outlets have been used. Each of these approaches to the problem is helpful, but each is rejected as inadequate. The fifth paragraph suggests what is needed to overcome the problem, and the sixth paragraph outlines a new and previously untested approach to overcoming the problem. 3 Example of a ResearchProblem Identification and Definition Statement: Africa l In Burkina Faso, a program of home visits by volunteers was initiated several years ago in Ouagadougou as one means of educating couples about family planning and referring them to appropriate service delivery sites. The program was expected to provide a possible model, particularly in urban areas, for the government's new family planning efforts. The use of volunteer, satisfied contraceptive acceptors to visit households in their neighborhoods and talk about family planning and the availability of services was thought to be a more effective approach to reaching couples than either mass media or using full-time, paid sages-femmes (midwives). 9. Although some of the volunteers have been quite active in recruiting new family planning clients, on the whole the program has not been as successful as originally anticipated. A diagnostic study of the volunteer program indicated several difficulties: a While the use of volunteers to make home visits is obviously less costly than the use of full-time, paid sages-femmes, it is not necessarily as effective. Volunteers may or may not be motivated to visit homes on a regular basis. Typically, initial motivation andenthusiasm quickly wanes and after a period of some months the volunteers visit few, if any, homes. b The area assigned to each volunteer, usually a Sector, is too large for one person to cover adequately. The volunteers tend to use an ad hoc method for making home visits and do not have a coordinated and scheduled plan. e There is little or no supervision of the volunteers in the field. This may be one reason that the enthusiasm of many volunteers declines. d Although the volunteers received an initial orientation course to family planning, no efforts are made to provide them with on-going training which might serve to strengthen and reinforce their skills. e Reports from couples contacted .by the volunteers suggest that the information provided during home visits tends to be superficial. Many volunteers apparently are unable to give the type of specific information about contraceptive methods desired by potential users. 9 f There is a lack of follow-up. Once a home has been visited, there is hardly ever a repeat visit. 3 These problems have been discussed extensively by the concerned health and family planning officials. It is not known which one or combination of problems m'e affecting the volunteer home visit program. On the basis of the diagnostic study, however, several possible solutions to the problems appear possible: a It might be possible to eliminate the volunteer workers entirely and replace them with paid, full-time sages-femmes. The sages-femmes would undoubtedly devote more time to home visits and would be able to provide more detailed information to potential clients. On the other hand, using paid sages-femmes would result in an extremely costly home visit program which probably would not be sustain- able. Also, it is unlikely that the necessary number of sages-femmes could be recruited because relatively few have been trained. b A second alternative might be to retrain the volunteers. While this solution would certainly be less costly than hiring full-time sages-femmes, it is not likely to solve all the problems. The area assigned to the volunteers would still be too large to be covered effectively. There would still be a need to institute a program of field supervision. Follow-up of clients would probably remain a problem, and even though retraining of the volunteers might improve their knowledge of family planning, it is unlikely that they would develop the type of detailed information possessed by the sages-femmes. e A third alternative would be to eliminate the home visit program entirely and rely on mass media campaigns and Sectoral community meetings as a means of educating people about family planning. While this approach might help to inform people, it is not likely to be entirely successful in changing their behavior. The literature on persuasive communication strongly suggests that behavior change is most often the result of interpersonal communication, not mass media. Thus this approach would probably not result in a large number of new family planning acceptors. d A fourth alternative would be to combine some of the best aspects of all three approaches described above. For example, a program could be designed 1) to assign volunteer, satisfied acceptors to a smaller and more manageable area that they could cover easily; 2) to have them work in a coordinated and scheduled manner for a limited amount of time each week with paid sages-femmes; 3) to provide volunteers with support, supervision, and periodic retraining; and 4) to organize Sectoral group meetings on family planning. The volunteers could help to identify friends, relatives, and neighbors most interested in services. The sages- femmes could supervise the volunteers and provide the more detailed, specific information on family planning and maternal child health to potential clients. 4 Comments on the Example The first paragraph focuses on just one country, Burkina Faso, and on a program in just one city, Ouagadougou, which tried to provide more available and accessible education and services to couples through the use of volunteers. This program was originally thought to be more effective than the use of mass media or full-time, paid sages-femmes. The second paragraph notes a problem situation, a discrepancy between what was expected (a well-functioning volunteer program) and what is (a program with problems). A recent diagnostic study of the volunteer program turned up six problems with the home visit program. The third paragraph notes that the problems with the home visit program have been discussed extensively but it is not known which one or combination of problems are the most important. Several possible solutions for improving the program are then presented. The first three are rejected as being inappropriate for one reason or another. The fourth solution, a combination of some elements of the other three, is accepted as the most likely to succeed and thus is the solution to be tested in an operations research intervention study. 10 What To DomProblem Identification and Definition 1 Follow this general procedure in identifying and defining a problem situation: a Start with a simple statement of the problem situation. b Add details as you review the literature and investigate the problem in greater depth. e Move back to simplicity by focusing on the most important aspects of the problem that are researchable. 2 Make a first attempt at identifying the problem situation by using the following format: Problem Situation: (Write a small, simple paragraph that identifies the problem) Discrepancy: (State the discrepancy between what is and what should be) Problem Question: (Write the central problem question) Possible Answers: (Write two or more plausible answers to the problem question) 3 From available research literature, current service statistics, educated opinions, and other sources of information, try to add detail to the problem situation you have just identified. Look for theoretical concepts and operational variables that you may have missed. List these concepts and variables on a piece of paper as you come across them. Try to answer the following questions: a What is the incidence and prevalence of the problem? b What geographic areas are affected by the problem? e What population groups are affected by the problem? d What are the findings of other research studies? e What has been done to overcome the problem in the past? f How successful were past efforts to overcome the problem? g What seem to be major unanswered questions about the problem? 4 With the information you have collected from a literature review and other sources, rewrite your statement identifying and defining the problem. Use the format of Problem Situation, Discrepancy, Problem Question, and Possible Answers. Add details that help to define the problem, but organize the information. Try to establish the boundaries of the problem. Focus your attention on the most important, researchable aspects of the problem. Be clear and specific. Have one or more friends read your final statement identifying and defining the problem situation. Have the friend tell you what he or she thinks the problem is. If your friend is unclear about the problem situation, if your friend cannot describe the discrepancy between what is and what should be, then go back to the beginning and start all over again! C Problem Justification Now that you have identified and defined the problem situation, it is necessary next to justify the importance of the problem. Research often is expensive and time consuming. Ask yourself, why is the problem I wish to study important? Can you justify your selection of the research problem? Can you convince others about the importance of the problem? I ExampleJustifyingthe Selectionof a ResearchProblem: Asia The difference between urban and rural contraceptive prevalence rates in Indonesia relates directly to the larger issues of rapid urban growth and to inadequate and ineffective service delivery to urban residents. Throughout South and Southeast Asia, urban population growth rates far exceed those in rural areas. Typically, urban areas grow at an annual rate of about 4-8 percent, 11 while rural areas grow at a rate of 2-3 percent. Most of the urban growth is due to migration from rural areas, but as the population base of cities expands, fertility also contributes to the growth. While migrants come in search of jobs and a better life, often they find that health, housing, and sanitary conditions in cities are worse than what they left behind. In most large cities, effective means to deliver basic health, and other services have not been found. The relatively low contraceptive prevalence rate in Jakarta compared with rural areas is a problem recognized by the government family planning program (BKKBN) and top government policymakers. This problem is an indication that health and family planning service delivery to urban residents is inadequate and ineffective. New models of service delivery based on the special situation of metropolitan areas and the life-style of urban residents must be found and tested. Particularly important is the need to reach new migrants to the city and those who are in the lower socioeconomic strata. For many of these people, health and family planning services are neither available nor accessible. So long as this situation continues, there will be significant differences in contraceptive practice between Urban and rural areas. 2 Comments on the Example The justification for selecting the research problem is presented in several different ways. First, it is pointed out that the problem of differences between urban and rural contraceptive prevalence rates affects many areas in South and Southeast Asia and is not just restricted to Indonesia and Jakarta. In other words, the problem is widespread. Second, this is a current and timely problem that is likely to become worse as the population base of cities continues to expand. Third, the problem is recognized as an issue of great concern by the BKKBN officials. Fourth, the problem relates to ongoing service delivery efforts, in particular to the inadequate and ineffective health and se_/ice delivery efforts in urban areas. Finally, the research justification statement suggests that, if new urban-based models for service delivery are not found, the contraceptive prevalence rate will continue to be lower in urban areas than in rural areas. 3 Example Justifying the Selection of a Research Problem: Africa Despite rapid urban growth in almost all countries of Africa, many family planning programs have paid less attention to urban centers than to rural areas. Moreover, those programs that have been initiated in cities have not always been successful. While urban residents tend to be better educated and have greater resources than rural residents, it is often mistakenly assumed that this implies they also have greater knowledge of and access to health and family planning facilities. In Burkina Faso, the urban poor are growing in numbers, not declining. New and innova- tive approaches to reaching these people are urgently needed. It is simply not realistic to expect that a few hospitals and clinics in Ouagadougou, most of which are overwhelmed with patients and geared toward curative medicine, can provide quality family planning education and ser- vices. It is also not realistic to assume that urban residents are any more willing than rural residents to spend time, money, and effort travelling to clinic centers where services may or may not be available. Education and services need to be taken to people. Just as community-based distribution programs have proved effective and sustainable in rural areas, Similar neighborhood- based programs need to be developed in urban areas. 4 Comments on the Example The problem justification statement points out that family planning programs in many African countries tend to ignore urban centers. In other words, the problem of inadequate family planning services for urban residents, a special and important population group, is widespread and not limited to a single country. The problem is also current and timely in Burkina Faso where the urban poor are growing in numbers, not declining. Finally, the research justification state- ment implicitly raises a larger issue of equity. Just as attention has been directed at developing CBD programs in rural areas, similar attention needs to be devoted to neighborhood programs in urban areas. 12 What To Do--Justifying the Selection of a Research Problem 1 In justifying the importance of a research problem, it is helpful to ask yourself a series of questions and then try to answer each of them. a Is the problem you wish to study a current and timely one? Does the problem exist now? b How widespread is the problem? Are many areas and many people affected by the problem? c Does the problem affect special groups, such as mothers or children? d Does the problem relate to ongoing program activities? e Does the problem relate to broad social, economic, and health issues, such as unemploy- ment, income maldistribution, the status of women, education, and maternal and child health? f Who else is concerned about the problem? Are top government officials concerned? Are medical doctors or other professionals concerned? ! 2 Review your answers to these questions, and arrange them into one or two paragraphs that justify the importance of the research problem. 13 Strategy Selectionto Solve the Problem A Usinga Quality-of-Care Framework Once an operations research problem has been identified, defined, and justified, a range of potential strategies for solving the problem can be suggested. These strategies often involve testing a modification to an existing service delivery or management approach rather than developing entirely new service delivery structures. It is not unusual to find that many program problems, and thus the potential strategies for solving these problems, are concerned with quality-of-care issues. In developing an appropriate strategy for solving a specific program problem, it is always helpful to refer to the six quality-of- care elements listed below, since each may suggest a solution strategy: I Broaden the range of contraceptive methods available to clients at clinics, hospitals, commercial, and community-based outlets. 2 Provide clear and specific information to clients about the different methods available and their advantages and disadvantages. 3 Improve the technical competence of service providers. 4 Enhance the quantity and quality of client-provider interpersonal relations. 5 Promote continuity of contraceptive use. 6 Provide an appropriate constellation of services to clients. B Examples of Strategies to Test ° Clearly, the specific strategies for dealing with the elements of quality of care or other operational family planning problems are vast in number. For example, among the hundreds of operations research studies conducted in Asia, Latin America, and Africa, strategies have been developed and tested for: • Increasing family planning service availability and access by using commercial outlets, work-based sites, satellite clinics, village contraceptive depot holders, mobile clinics, client transportation assistance, and flexible service delivery hours. • Improving information, education and communication by linking mass media with more interpersonal channels of communication such as adult education classes, village group meetings, client counselling, peer counsellors, and satisfied acceptors. • Introducing or promoting particular contraceptive methods by using alternative types of personnel, such as nurses for IUD or NORPLANT _ insertions; establishing male clinics to promote vasectomy; conducting education meetings for mothers' clubs to promote breastfeeding and natural family planning. • Improving worker performance through better selection procedures and criteria, competency-based training, realistic job descriptions, better or more selective supervi- sion, and an improved work environment. • Strengthening program performance through management training, better flow of information for decision-making and progra m planning, use of micro computers, im- proved completeness and accuracy of service statistics, systems for expediting the flow of 14 financial resources, more effective and efficient logistics systems, and better maintenance of facilities, vehicles and other equipment. _ • Meeting the needs of new target groups such as males, adolescents, women in the work place, urban poor, ethnic or religious groups by targeting IEC materials and services to these groups. • Increasing cost recovery and program sustainability through private sector involve- ment, fees for products or services, commercial social marketing, employer contributions for services, local government contributions, community insurance schemes, and public and private sector joint ventures. • Integrating family planning services with other health programs such as vaccination, breastfeeding promotion, and other primary health care activities. C Guidelines for Selecting an Appropriate Strategy to Test. Before selecting a management or service delivery strategy to test in an operations researeh study, first go back and review again the details of the program problem that concerns you. The nature of the problem should determine the selection of a test strategy to overcome the problem. In other words, an appropriately selected strategy is one that solves a program problem. This may sound like a rather obvious point, but it is surprising how often a single strategy such as social marketing or the use of more IEC (posters, videos, brochures and pamphlets) is viewed as a panacea and is repeatedly applied to just about any program problem whether 0r not it is an appropriate strategy. In addition to carefully reviewing the program problem, there are other guidelines you can follow to help select and develop an appropriate strategy to test in an operations research study. 1 Review the strategies other people have used to solve similar program problems. One way to do this is to examine the operations research findings from the many family planning studies conducted in the past. Find out what management and service deliv- ery strategies seem to work best for particular program problems. 2 Hold a meeting with the people who are most affected by the problem or most concerned with it. People who ate affected by management and service delivery problems usually have opinions, often strongly held, about possible solution strategies that can be used to overcome the problem. It is always important to obtain their ideas since more often than not they can come up with better strategies than anyone else. 3 Look for strategies that can be implemented without overburdening the imple- menting institution. If the resource requirements for testing a solution strategy are so high that they interfere with current service delivery efforts, then the strategy should not be implemented. 4 Select strategies that can be sustained over time. No matter how effective apa, ticu- lar strategy is in the short run, that is, during the intervention phase of an operations research study, if it cannot be sustained by an organization in the long run (after the OR study is completed) it is not likely to have much effect on overcoming a program problem. Always seek strategies that have a high probability of being financially sustain- able. 5 Seek strategies that are simple to implement. In deciding between alternative strate- gies of service delivery with potentially equal impacts, select the strateg_y that is the simplest in terms of facilities and equipment needed, personnel and training required, time and money spent, logistics and management required, and other similar consid- erations. 6 Develop strategies where the proposed solution is under the control of program managers. A solution strategy, or in the language of research, "the independent vari- ables," must be under the control of family planning managers. 7 Avoid strategies where the cost of a field test is higher than the expected benefits. Clearly an operations research strategy should not be tested if the expected benefits are trivial relative to the implementation effort required. 8 Avoid strategies that are not consistent with the implementing institution's goals, objectives, and development plans. A strategy that does not fall within the current goals, object|yes and plans of an implementing agency may meet with considerable 15 resistance and not receive the resources and attention required for successful imple- mentation. I Example of a Strategy to Solve the Problem: Latin America The relatively high percent of oral contraceptive users who do not know how to take the pill correctly is a problem that relates directly to at least two quality-of-care elements, namely: 1) information provided to clients; and 2) client-provider interpersonal relations. When 58 percent of oral contraceptive clients take the pill incorrectly, it seems fairly clear that they are not receiving adequate information. Either they are not being given this information by providers, or a negative client-provider relationship results in ineffective communication of information. This situation suggests several potential strategies for overcoming the problem. First, IEC materials such as brochures, posters, and a video could be developed on oral contraceptive use and an effort could be made to reach all new and existing clients with these materials. A second possible strategy would be to revise the current training program of the rural health promoters. In particular, the existing rural health promoters could be re-trained with an emphasis placed on developing effective communication skills necessary to provide clients with full information on all aspects of oral contraceptive use. It is this second strategy that will be developed, tested, and evaluated first before other alternative strategies are considered. The primary reasons, for selecting a retraining strategy are the following: a Previous operations research studies in Latin America strongly suggest that interperson- al communication between clients and providers that takes place in an atmosphere of understanding, respect, and honesty, and which allows for questions and answers, tends to be a more effective means of conveying information to clients than impersonal, mass media approaches. b The family planning program already has an existing rural health promoter training and retraining program. It will be far easier to revise the current curriculum and over a period of 18 months provide retraining for all rural health promoters than to develop, pretest, and produce new IEC materials. c The retraining strategy is a more sustainable approach because both the initial and subsequent long-term costs are far less than an IEC materials development strategy. 2 Comments on the Example • In this example, the problem of incorrect pill use is related to quality-of-care elements. These elements suggest two possible strategies for overcoming the problem situation. One would provide clients with information on pill use via mass media channels (brochures, posters, videos), and the other would rely on interpersonal channels of communication between provider and client after rural health promoters have been retrained. The second strategy is selected as the one that will be developed, field-tested, and evaluated. Three reasons for selecting this strategy are given: 1) previous OR studies suggest it is more effective, 2) retraining will be simpler and easier than developing new IEC materials, and 3) the retraining strategy is less costly and thus more sustainable in the long run. 16 What To Do--Selecting a Strategy to Test After a significant program research problem has been identified, defined, and justified, you will need to select an appropriate strategy to solve the problem. First, review the nature of the program problem that concerns you. Does the problem relate to the larger quality-of-care issue, and if so, are one or more strategies suggested by the six quality-of-care elements? Refer to the guidelines for selecting a strategy and then in your research proposal you should: 1 Identify potential strategies that could be used to solve the program problem. 2 Clearly indicate which one or more of the potential strategies you have selected as the most appropriate to solve the program problem. 3 Justify your selection of one or more potential strategies by indicating: a Past success with applying the strategy to similar problem situations. b The simplicity of implementing the strategy over other strategies that are likely to produce the same impact. c The potential for sustaining the strategy once the operations research study is completed. 17 Objectives and Hypotheses The objectives and hypotheses of a research study should-flow logically from the earlier sections identifying the problem situation, defining the parameters of the problem, and justifying its importance. In this section, the research purpose is narrowed and focused. Specific objectives are written that describe the expected contributions and outcomes arising from the study and the variables that will be measured. Also, specific, testable hypotheses ate formulated that give the relationship between variables. A Ultimate Objectives Most research studies have a statement of ultimate objectives that describes the expected implications or contributions arising from the study. The ultimate objective of many health and family planning studies is to contribute in some way toward reducing morbidity, mortality, or fertility. This of course is a rather unrealistic objective. It is unlikely that any Single study will do much to reduce morbidity, mortality, or fertility. A somewhat more reasonable ultimate objective is that the study will provide program administrators and policymakers with information useful for improving service delivery activities or reformulating policy. Ultimate objectives help to relate the proposed research to broad social, economic, and health concerns. They state the reasonable and expected contributions of the study. In this way, they contribute to the justification of why research on the problem situation is required. Note, however, that ultimate objectives are expected and hoped for contributions. In a sense, they represent wishful thinking. The investigator does not promise that the contributions will occur. 1 Examples of Ultimate Objectives 1. Study of Areal Variation in Contraceptive Practice in Bangladesh. "The ultimate.objective of this study is to help program administrators to design new educational strategies for motivating couples in villages with low contraceptive prevalence." 2 Study in Burkina Faso to Test an Approach to Providing Family Planning Services to Urban Residents. "The ultimate objective of this study is to develop a cost effective and sustainable model for providing quality health and family planning education and services to urban residents." 3 An Experimental Study to Integrate Community-based Family Planning Education and Services with Primary Health Care in Two Rural Areas of Cameroon. "The ultimate objective of this study is to develop a rural, integrated, cost-effective family planning and health services svstem which has a measurable impact on decreasing infant and mater- nal morbidity and mortality." 4 Survey Study of the Family Planning Knowledge, Attitudes, and Practices of Community Influentials in Sri Lanka. "The ultimate objective of this study is to increase the involve- ment of community influentials in health and family planning program activities." 18 B Immediate Objectives In contrast to ultimate objectives that state what is expected to happen, immediate objectives state what will happen. Immediate objectives relate directly to the research problem situation. They indicate the variables that will be examined and measured. An imlnediate objective repre- sents a promise by the investigator that certain activities will t',_e place and specific variables will be examined. Whenever possible, and certainly in all evaluation studies, the immediate objective should be stated in behavioral terms--in other words, in terms of what will be done. The objective should specify: Who will do, how much of what, to whom, when, where, and for what purpose. 1 Examples of Immediate Objectives 1 An Experimental Operations Research Study in The Gambia to Provide Family Planning Services at the Work Place. "Over an 18-month study intervention period, The Gambia Family Planning Association, working with 15-20 selected factories, hotels, and other enterprises in the Banjul area, will implement an operations research study to test two models of service delivery at the work place. The two approaches will be compared against each other and against a third (control) model in terms of cost, knowledge of contraception, use of contraception, satisfaction with services, and other measures." 2 Study in Indonesia of Urban and Rural Differences in Contraceptive Prevalence Rates. "In the first phase of this study, the Indonesian BKKBN will conduct in 1991 an exploratory study in Jakarta and in two rural Kecamatan to determine the institutional and program-related factors associated with the variance in contraceptive practice among currently married women aged 15-44. "In the second phase of this study, to be conducted in 1993, the BKKBN will evaluate a new model of health and family planning service delivery in temps of (a) increased contraceptive practice among currently married women aged 15-44, and (b) greater utilization of primary health care facilities among people in the lower" socioeconomic strata." 3 Evaluation of Different Training Methodologies. "The Training Division of the Family Planning and Maternal and Child Health Project in Nepal will conduct an evaluation study in 1991 to determine the difference in field-worker performance between those workers who received a classroom-based training plx)gram and those who received a field-based training program." 4 Survey Study of the Family Planning Knowledge, Attitudes, and Practices of Community Influentials in Sri Lanka. "In 1992, the Ministry of Plan Implementation will conduct a survey in 100 randomly selected villages to determine the family planning knowledge, attitudes, and practices of community influentials." 2 Comments on the Examples Note that, in each of these objectives, the investigator" promises to undertake a specific behav- ioral activity: to conduct, to evaluate, to survey, or to determine. The situation or context of the research is also specified. Who will conduct the study where and when is stated. Finally, and most important, the key variables of interest are indicated--social, economic, institutional, and program factors; contraceptive practice; utilization of primary health care facilities; field-worker performance; training; and so on. Each of these variables will need to be carefully defined in operational terms later, but at least as stated here in the immediate objectives, they serve to indicate the focus of the proposed research. C Hypotheses A hypothesis is a statement about an expected relationship between two or more variables that permits empirical testing. While ultimate objectives give the anticipated contributions arising from a study and immediate objectives (in behavioral terms) what willbe done or measured, 19 hypotheses specify the expected relationship among the variables. Hypothesis statements are most appropriate for field intervention or evaluative studies. Diagnostic or exploratory studies do not normally require hypothesis statements because they generally do not test relationships between variables. Study hypotheses serve to direct and guide the research. They indicate the major indepen- dent and dependent variables of interest. They suggest the type of data that must be collected and the type of analysis that must be conducted in order to measure the relationship. A single hypothesis might state that variable A is associated with variable B, or that variable A causes variable B. Sometimes a hypothesis will specify that, under conditions X_Y, and Z, variable A is associated with or causes variable B. A well-written hypothesis focuses the attention of the researcher on specific variables. When writing hypothesis statements, it is important to keep in mind the distinction between independent and dependent variables. An independent variable causes or deter- mines or influences the dependent variable• The basic relationship between these two types of variables can be represented as in Figure 4.1• This model shows a direct relationship. In other words, whenever the independent variable changes, there is a change in the dependent variable• The dependent variable depends on the independent variable. In most family planning operations research studies, it is unusual to find a direct relation- ship of this type. Almost always, the relationship is indirect. The independent variable acts upon the dependent variable through intervening variables. These intervening variables serve either to increase or to decrease the effect the independent variable has on the dependent variable. Intervening variables are sometimes referred to as test or control variables. Many, if not most, family planning research studies are based on a model that includes intervening variables (see Figure 42,). For example, in an effort to increase contraceptive acceptance (the dependent variable), a family planning program might initiate a large information, education, and communication (IEC) program (the independent variable). Alone and by itself, an IEC program cannot increase contra- ceptive use. The program must act on and operate through a set of intervening variables, which in turn cause or determine or influence contraceptive use. There may be many of these intervening variables, but the most likely ones that would be influenced by an IEC program would be people's knowledge about family planning, their attitudes toward the use of birth control methods, and their beliefs about family size. A possible research model for an evaluation study of the effects of an IEC program is shown in Figure 4.3. I I IA direct relationship Independent Causeordetermineorinflu Dependentbehveen independent and variables variables dependent variables. FIGURE 4.2 I ] Intervening variables Interveningvariables I Ivariables variables 2O FIGURE 4.3 [ I The effect of intervening People'sknowledge, variables in an IEC program attitudes, beliefs [ [ Cootracept,ve[program acceptance In family planning operations research, the hypotheses of most importance usually are those that consider program activities as the independent variable. The family planning input, such as an IEC effort, is the independent variable. The objective of the research is to determine the extent to which the independent variable affects the dependent variable. The dependent variable is usually the central concern of a research proposal's problem statement. The interven- ing variables are important because they tend either to increase or to decrease the strength of the relationship between the program (the independent variable) and the outcome effect (the depen- dent variable). Most research studies will examine many independent variables and many intervening variables but only a few dependent variables. In writing study hypotheses, always think in terms of the expected relationship between variables. Think first about the central problem your study will address (the dependent variable). Next consider what factor or factors (the independent variables) might cause, determine, or influence the problem situation. Finally, ask yourself if the relationship between the independent and dependent variables is direct or indirect through a set of intervening variables. 1 Examples of Hypotheses 1 Villages with many modem characteristics x__ll have a higher rate of contraceptive practice than villages with few modem characteristics. 2 Contraceptive practice will be higher in villages where the family planning field-worker. is viewed as credible than in villages where the field-worker is not viewed as credible. 3 The work performance of health and family planning staff who have received a five- week, field-based training course will be higher than the work performance of similar staff who have received a four-week, classroom-based training course. 4 Higher levels of field-worker performance will lead to higher rates of contraceptive acceptance among eligible couples. 5 An information and education program using radio, television, and printed material will increase people's knowledge about family planning. 6 People with higher levels of family planning knowledge will be more likely to be users of contraceptive methods than people who are not as knowledgeable about family plan- n_g. 2 Comments on the Examples Note that in each hypothesis there is a statement of an expected relationship between two or more variables. In the first hypothesis, the expected relationship is between modem characteris- tics of villages (the independent variable) and contraceptive practice (the dependent variable). In the second hypothesis, the relationship is between the credibility of field-workers (the indepen- dent variable) and contraceptive practice (the dependent variable). The third and fourth hypothe- 21 FIGURE 4.4 I J A model for two hypotheses Field-workerpedormance about the effect of a field- worker training program on c°tnetsraCeptive acceptance .,.,,_ _o_ __ I I I've Itrainingprogram acceptancerates ses go together. In the third hypothesis, the relationship is between a five-week, field-based training program (the independent variable) and field-worker performance (the intervening but also dependent variable). In the fourth hypothesis, field-worker performance acts as an indepen- dent variable and contraceptive acceptance rates become the dependent variable. Taken together, the model for these two hypotheses would look like Figure 4.4. Similarly, hypotheses five and six go together. Family planning knowledge is the dependent variable in hypothesis five, but becomes the independent variable in hypothesis six. To summarize, all research studies should include a statement of (1) ultimate objectives, (2) immediate objectives, and (3) hypotheses (except in the case of purely exploratory or descriptive studies). Each of these statements serves a different purpose: ] Ultimate objectives state the anticipated (hoped for) contributions of the study. _- Immediate objectives state what will be done in the study. 3 flypoiheses state the expected relationship between two or more variables. What To DomWriting Ultimate Objectives, Immediate Objectives, and Hypotheses 1 List the major variables of your study under the headings: Independent Variables and Depen- dent Variables. Intervening variables can be listed under both headings. The dependent variable(s) should relate directly to your problem statement. 2 Review the list of dependent variables, and then write a statement of ultimate objectives that relates to them. Ask yourself the following question: "If I knew the factors that caused, determined, or influenced the dependent variable(s), how would this knowledge help policy- makers, program administrators, or others?" How will the results from the study help improve service delivery, improve training programs, assist in the design of educational materials? In other words, what are the anticipated contributions of the study? 3 Now write the immediate objectives for your study. Focus your attention on the specific actions that you will perform. Exactly what do you plan to do? What variables do you plan to measure? Write behavioral objectives that answer the following questions: a Who will do? b How much of what? e To whom? d When? e Where? f For what purpose? 4 Finally, write the hypotheses that your study will test. Review once again your list of indepen- dent and dependent variables. How are these variables related? Are there special conditions that must be present before they are related? Write hypothesis statements in positive, not negative, terms. Write a hypothesis for each major relationship that you expect to test in your study. 22 Intervention Description If your study includes an intervention, such as a test of a new training procedure or a comparison of two new approaches to service delivery, you will need to describe the nature of the intervention fully in your study proposal. Essentially, you should elaborate on the questions you answered when writing the immediate objectives--who will do, how much of what, to whom, when, where, and for what purpose. More specifically, it is important to answer in some detail the following questions: A Who Will Be Responsible for Implementing the Intervention? Indicate not only the organization(s) that will be responsible for implementation, but also the categories of people (nurses, doctors, midwives, field motivators, etc.) who will be involved and the role of each person. n !) Where Will the Intervention Activities Take Place? Be as specific as you can about the actual location of the study intervention. Will the intervention be implemented in an entire province, in one district, in ten villages, in thirty factories, or in four clinics? Be sure to mention the location of the study sites. (. What Activities Will Be Initiated? You should describethe sequence of events that will take place. For example, a study intervention might start with the training of field workers. The nature and duration of the training should be described. Next, a series of village meetings might be held. The frequency and purpose of these meetings should be noted. Finally, the field workers will visit at least five homes each day for a period of one year. The purpose of the visits and the expected activities during the visits should be described. In describing the activities that will be implemented, it is important to indicate the level or intensity of each activity. For example, if the intervention being tested is an IEC campaign, you should describe in detail the nature of the campaign. Indicate the type of media that will be used (television, radio, newspapers, pamphlets, wall posters, .group meetings, plays, songs); mention the intensity of the media effort (100 wall posters, 1,000 pamphlets, three radio spots per day for six months, one TV program a week for three months); specify who will produce the posters, pamphlets, radio and TV scripts, plays and songs, what kind of training (if any) will be needed, and what special equipment will be required. In short, be detailed and specific. 1 Exampleof a StudyIntervention Description: Africa An Experimental Operations Research Study in The Gambia to Provide Family Planning Services at the Work Place 23 2 Study Intervention Sites This employment-based family planning study will be implemented by The Gambia Family Planning Association (GFPA). All employment sites will be loc0ted in and around Banjul. Potential sites will include those with at least 100 employees of whom 20 or more are female workers. It is estimated that some 25 employment sites will meet these criteria and approximately 15 to 20 of these will agree to participate in the project. Thus, at a minimum, the 15 employment sites should have approximately 1,500 employees (15 sites x 100 employees) of whom 300 (1,500 x 20) should be female employees. In fact, the number of total employees and female employees is expected to be far greater. Although The Gambia does not have large industrial enterprises, there are several that employ well over 100 workers, such as breweries, hotels, the port authority, and commercial farm operations. In some of these enterprises the majority of workers are female. For example, a shrimp cleaning and packing plant has more than 200 female employees. The GFPA has already contacted a number of work sites to determine potential interest in participating in the study, and all enterprises contacted have expressed an interest. In order to ensure active and informed participation in the study, site visits and seminars will be organized by the GFPA for management and workers' representatives before the study intervention begins. The purpose of these seminars will be to outline the project design and objectives and to convince management of the short- and long-term advantages of participating in the activity. The seminars will be organized in collaboration with the Ministry of Labor and the Department of Medicine and Health. 3 Study Activities Employment places willing to participate in the project will be stratified on the basis of the number of female employees. Two strata will be created: 1) employment sites with 20-49 female employees, and 2) sites with 50 or more female employees. From within each stratum, employ- ment sites will be randomly assigned to one of the three groups---stationary services, mobile services, and the control group. The activities to be initiated in the two experimental groups are described below: a Stationary Service Sites At the stationary service site, one family planning distributor for every 40 workers will be selected by the workers themselves, with the approval of management. The distributors will receive a total of ten days of training on family planning and sexually transmitted diseases (STDs). Five days of Lraining will be completed before the intervention phase of the study begins. The remaining five days of training will be spread over the life of the intervention period, one day at a time. The primary job of the distributor will be to provide fellow workers with information on family planning and STDs, to sell condoms and spermicides on a commission basis, and to make referrals to the GFPA clinics for contraceptive methods such as the IUD and sterilization. Each distributor will be given three bright colored T-shirts with the GFPA logo. This will help other workers identify them. In addition, they will be given a bag to store contraceptives and record- keeping forms. The stationary service sites will receive monthly visits from a GFPA educator, who is a trained midwife. One of the primary functions of the GFPA educator/midwife will be to sell the distributor condoms and spermicides for re-sale to clients. The distributor will be allowed to keep half of the final sale price. Currently, a strip of four condoms sells for approximately 12 cents. Thus for each strip of four condoms sold, the distributor earns 6 cents and GFPA recovers 6 cents. The GFPA educator/midwife will examine women who desire pills and, ff appropriate, provide them with an initial cycle and re-supplies during later visits. While the pills will be supplied to clients free, the educator/midwife will charge each pill .user a small service fee. This is the same procedure used at the GFPA clinics. In addition to re-supplying the distributors and providing pills to new and continuing users, the educator/midwife also will hold group meetings with the workers to discuss STDs, family planning methods, side effects, and sources of supply. Work-site managers will be re- quested t° allow 15 minutes of work time every two weeks for educational sessions. The workers will be asked to give an equal amount of time for these educational sessions during their work break. 24 b Mobile Service Sites Work sites assigned to the mobile services group will receive visits every two weeks by a GFPA educator/midwife. She will hold group meetings with the workers to discuss family planning methods, side effects, and sources of contraceptive supply. Work-site managers will be requested to allow 15 minutes of work time every two weeks for educational sessions. The workers will be asked to give an equal amount of time for these educational sessions during their work break. The meetings will also provide information on STDs including AIDS (HIV). In addition, the educator/midwife will sell condoms and spermicides to acceptors. The full price of the sale will be returned to GFPA. As in the stationary service sites, pills will be provided free, but a small service fee will be charged. For other methods such as the IUD and sterilization, the educator/midwife will make an appointment for the worker at the nearest GFPA clinic. 4 Recruitmentand Trainingof Field Staff The GFPA currently has a cadre of educator/midwives who are responsible for carrying out the Association's information, education and communication programs. From this cadre, four will be selected to work on this OR study. They will be responsible for conducting the field activities, scheduling the on-site education visits, monitoring the contraceptive distribution activities, collecting routine se_ice statistics on sales and new and continuing users, and providing in- service education and training for the work-site distributors. Two educator/midwives will be responsible for the activities in the stationary service group, and the other two will be responsible for the mobile service group. Although the educator/midwives have already been trained in the substantive areas of family planning, a short orientation program on this OR study will be organized for them. This orientation will provide an opportunity to discuss the design of the project and to review the requirements and responsibilities of the GFPA field staff and work-place distributors. An existing member of the GFPA staff will serve as a research assistant. The research assistant will interview workers and collect service statistics and other relevant data from the work places. Training for the research assistant will be provided by the GFPA office. Recruitment of the work-place distributors will be coordinated by the Principal Investiga- tor with assistance from the two educator/midwives. Employees in the work sites will be encour- aged to participate in the selection of the distributor for their establishment. They will be provided with selection criteria to assist them in identifying suitable distributors. The distributors selected should show an interest in becoming involved in the project, demonstrate positive attitudes toward contraceptive use, and be respected by workers and management. Distributors will be required to participate in an initial five-day training session and in a subsequent series of five one-day training sessions to be conducted by the staff of GFPA. These training sessions will be scheduled, on days that are convenient for the participants and trainers. What To Do--Intervention Description 1 Be as detailed and complete as possible in describing the study intervention. 2 Describe the activities in the order they will occur. 3 Be sure your description of the intervention answers the three basic questions: a Who will be responsible for implementation? b Where will the intervention take place? e What activities will be initiated at what level of intensity? 25 Operational Definitions After formulating the study objectives and hypotheses, and describing fully the study interven- tion, the next step in the research process is to define operationally the key variables and terms of the study. Operational definitions serve two very essential purposes: 1) They establish the rules and procedures the research investigator will use to measure variables; and 2) they provide unambiguous meaning to terms that otherwise can be interpreted in different ways. Every research proposal must include operational definitions of major variables and terms. A OperationaIDefinitions of Variables Suppose that a dependent variable of a study is family planniflg knowledge. Before this variable can be measured, it is necessary first to establish the operational procedures that specify how the measurement will be made and at the same time define what the researcher means by the three words family planning knowledge. The variable must be defined in terms of events that are observable by the senses. The observable events serve as an indicator of the variable, family planning knowledge. Alone and by itself, knowledge is not observable by the senses. You cannot touch knowledge, or see knowledge, or smell it, taste it, or hear it. What is needed is an observable event that indicates knowledge. Usually, such an indicator is based on a series of questions. For example, we might ask a respondent, "Have you ever heard about contraceptive pills? . Can you tell me how often a woman should take the contraceptive pill in order to avoid pregnancy?" "Have you ever heard about the IUD? . Can you tell me where a woman can get an IUD?" Each of these questions serves to indicate whether the respondent knows about certain aspects of family planning. Asking a question and hearing a response is an observable event that can be measured. A research study might ask 20 knowledge questions. Each time a respondent gives an answer that indicates knowledge, the researcher could record a score of I (one). Every time an answer is given that does not indicate knowledge, the researcher could give a score of 0 (zero). For each respondent, the researcher could add the total number of correct answers to the 20 questions and create a family planning knowledge score. This score would range from 0 correct answers to 20 Co_Tect answers. Persons with a score of 0 would be said to have no family planning knowledge. Persons with a score of 20 would be said to have high knowledge. What this procedure does is to define operationally the variable, family planning knowledge. In the research proposal, the variable might be defined operationally as follows: Family Planning Knowledge = The number of correct answers a respondent gives to 20 questions on family planning. Of course, this is not the only way the variable could be defined operationally. We might wish, for example, to establish categories of family planning knowledge, distinguishing between those respondents who have high family planning knowledge, medium knowledge, low knowl- edge, and no knowledge. Each of these levels is a category of the variable, and each requires an operational rule that tells us how to assign any given respondent to a category. One way of operationally defining the categories might be the following: 26 High Knowledge = Correct responses to 15 or .more of the 20 questions. Medium Knowledge = Correct responses to at least 8 but not more than 14 of the 20 questions. Low Knowledge = Correct responses to at least 1 hut not mole than 7 of the ques- tions. No Knowledge -- No correct answers to any of the 20 questions. Notice that the four categories of the variable are mutually exclusive (they do not overlap). By the operational rules established, a person cannot be placed in the category "High Knowledge" and at the same time be placed in the "Medium," or "Low," or "No" category. The categories are also totally inclusive. There are,only four categories. There is no fifth or sixth category that a respondent might possibly fit into. In some instances, we may not want to be quite so specific in defining before data collection the exact ways in which a variable will be split into categories. Sometimes it is preferable to determine the category "cutting points" after data have been collected and the response distribution for the variable can be examined. As a general rule, it is best to have approximately an equal number of respondents in each category. Thus, in the above example, each of the four categories of the variable family planning knowledge should have approximately 25 percent of the respondents in the study population. If it is necessary to examine the response distribution of a variable before the procedures for establishing categories can be determined, then in the operational definition section of a study proposal the category names can be specified, but with a note indicating that each category will comprise approximately equal numbers of respondents. All variables must have at least two or more categories, otherwise they are not variables but constants. Whenever you are operationally defining a variable, it is always better to divide the variable into many categories instead of just a few. In the examples given above, the variable family planning knowledge ranges from 0 to 20. That range gives a total of 21 separate categories. Subsequently, in the second example, we collapsed these 21 categories into just four--high, medium, low, and no knowledge. If we wanted to, we could go even further and collapse the four categories into just two: Knowledge of Family Planning --- A correct response to 1 or more of the 20 questions. No Knowledge of Family Planning = No correct answers to any of the 20 questions. If you start with many categories, it is always easy to collapse these down to just a few. But do not make the mistake of starting with just a few categories, because subsequently you cannot expand them into many. Collapsing the categories of a variable from many to just a few usually is done after data collection has been completed and the frequency distribution of the variable can be examined. I Examplesof OperationallyDefinedVariables Family Planning Acceptance -- The initiation of use of any modem method of contra- ception. Field-Worker Performance = The number of job-related tasks a field-worker under- takes out of 20 defined tasks. Improved Worker Performance -- Any government family planning field-worker who re- cruits more new family planning acceptors, holds more village meetings, or distributes more oral rehydration solution packets than he or she did six months earlier. Family Planning Attitude = The number of positive responses to 10 questions indi- cating approval of the use of contraceptive methods. B OperationaIDefinitions of Terms Recall that a hypothesis is a statement about an expected relationship between two or more variables. Just as it is necessary to define variables operationally, so too is it necessary to define 27 the terms that indicate the nature of the relationship between the variables. For example, in many hypothesis statements, you will find such terms as: more than greater than less than larger than higher than bigger than lower than smaller than You will also see in hypothesis statements such words as: safer significant acceptable expanded improved increased Each of these terms can have a variety of meanings, and thus each requires an operational definition for the research proposal. The basic" problem with such terms as mope than or Increased is that they denote a comparison but do not indicate the standard of comparison. We need to know how much more and increased by how much. Suppose a study has the following simple hypothesis: A five-week, field-based training program will increase the family planning knowledge of health and family planning workers who have taken the program. The training program is the independent variable. In the hypothesis, this variable is already defined, at least partially, as five weeks long and field-based. Family planning knowledge is the dependent variable. We already have defined this variable as the number of correct responses to 20 questions. What is left is to define the word increase. If we do not define this word, it will be impossible for us to know when the hypothesis has been proved or disproved. In other words, we need a standard of comparison that will tell us Increase by how much. One way to define Increase might be the following: Increase -- Among trainees, a mean family planning knowledge score on the posttraining test that is greater than and significantly different from the mean family planning knowledge score of a control group of field-workers who did not receive the training program. Note that this operational definition not only tells us the meaning of Increase but also gives us the procedures that will be used to measure the increase. The mean family planning knowledge score of trainees (in an experimental group) will be compared against the mean family planning knowledge score of a control group. The hypothesis will be accepted only if the trainees' mean score is greater than and significantly different from the mean score of the control group. Finally, to be absolutely clear, we also should define the word significantly: Significantly = A probability equal to or greater than .95 that the trainees' mean score is higher than the control group mean score. To summarize, operational definitions establish the rules and procedures an investigator plans to use to measure and give meaning to variables and terms. The definition is in terms of indicators that are observable events. We must be able to ask a question, hear a response, see a behavior, record an action, measure an attribute. The definition establishes categories for variables. The categories must be mutual-I F exclusive and totali F inclusive. Operational defini- tions also establish the standard of comparison the investigator will use to either accept or reject a hypothesis. 1 Examplesof Operationally Defined Terms Family Planning User = Any currently married woman aged 15-49 and any currently married male aged 15 and older who says he or she has used a modem method of contraception at least once during the last month. Modem Method of Contraception = Any one of the following pregnancy prevention'methods: pills, IUD, female sterilization, male sterilization, Depo-Provera, con- dom, diaphragm. 28 Modem Village = Any village that has three or more of the following facilities: electricity, a government health clinic, a paved road within half a mile, a primary school, a bank, a post office, irrigation for 50 percent or more of the farmland. What ToDo---Writing Operational Definitions I Write an operational definition for each variable on your list of independent and dependent variables. 2 Write an operational definition for each term used to connect the variables. 3 For each definition you write, ask yourself: a Are the rules and procedures for measuring the variables clear? b Have mutually exclusive and totally inclusive categories for the variables been established? 29 Study Design A Reliability and Validity The design of a study is the investigator's plan of action for answering the research question(s). The objective in selecting a study design is to minimize possible errors by maximizing the reliability and validity of the data. Reliability refers to the consistency, stability, or dependability of the data. Whenever an investigator measures a variable of interest, he or she wants to be sure that the measurement provides dependable and consistent results. A reliable measurement is one that if repeated a second time will give the same results as it did the first time. If the results are different, then the measurement is unreliable. Validity refers to data that are not only reliable but also true and accurate. Put another way, validity is the extent to which a measurement does what it is supposed to do. If a measurement is valid, it is also reliable. But if it is reliable, it may or may not be valid. For example, suppose an investigator asks a respondent, "How old are you?" The respondent replies, "I am 49 years old." The investigator then asks a second question to check on the consistency or dependability of the age measurement: "In what month and year were you born?" The respondent replies, "In September 1941." If it is now February 1991, the investigator calculates that indeed the respon- dent is 49 years old. In this example, two questions have been asked. Each is designed to determine how old the respondent is, and each question gives the same results. The results are consistent, stable, and dependable; they are reliable. But suppose later on the investigator happens to see the respondent's birth certificate, which shows that the true date of birth was September 1938. The investigator then concludes th:_t, although the first two questions gave reliable results, they did not give valid or true results. B Validity Threats In the selection of a research design, one criterion to use is the extent to which the design controls for threats to validity. In other words, a researcher wants a design that will give true and accurate information--a design that will avoid factors that might invalidate the study. While there are many such factors, some of the more common threats to validity are explained below. 1 History Sometimes events occur during the life of a project that tend either to increase or to decrease the expected outcomes of the project. Histol]y effects refer to these events. The events are not part of the project; they are not planned or anticipated events. They just happen, and they produce an effect that influences the study results. For example, suppose that a group of field-workers has been given.a special training course that is designed to improve their performance. Six months after the course, an evaluation is conducted, and the results indicate that there has been no improvement in field-worker perfor- mance compared with the pretraining period. The evaluator might conclude that the training 3O program was a failure. But a further examination of the situation reveals that soon after the training program was completed, all field-workers decided to go on strike for three weeks in an effort to raise their salaries. The strike represents a history threat to the validity of the evaluation. Obviously, the strike was not a planned or anticipated project event. It just happened, and the effect was to influence the performance of the field-workers. The question then arises, was the training program really a failure? If we could somehow control for the effect of the stlike, would an evaluation of the training program still reveal that the field-workers' performance did not improve? In other words, what is the true and valid effect of the training program? The ability to answer this question depends on the study design the evaluator has selected. In a second example, suppose that an evaluation is conducted to determine the effect on people's knowledge and attitudes of a family planning motivational campaign. A precampaign survey might be conducted, and the results then compared with the postcampaign survey results. The comparison might indicate substantial increases in people's knowledge about family plan- ning and more favorable attitudes toward birth control. Very likely the evaluator would conclude that the motivational campaign was a success and its objectives were fulfilled. But suppose that during the campaign the president of the country made a major radio and TV address to the nation on the subject of rapid population growth and the need for family planning. The question arises, was it the motivational campaign that produced the increases in knowledge and changes in attitudes, or was it the president's address, or possibly was it both? The president's address represents a history effect. Unless there is some way in the study design to control lot this effect, it is impossible to determine the true and valid results of the motivational campaign. In other words, the separate effect of the campaign cannot be distinguished fi-om the history effect of the president's speech. 2 Selection A very common threat to validity occurs whenever the people selected for the control group differ greatly from the people selected for the experimental group. For example, suppose that a special family planning incentive program is offered to people in one district (the experimental group) but not to people in another district (the control group). After a year, the incentive program is evaluated in terms of the family planning prevalence rate in both districts. The evaluation shows that there has been a significant increase in the experimental district over the control district. In this situation, the evaluator might conclude that the incentive program was a success: It in- creased the prevalence rate. But upon further checking, the evaluator discovers that there are large and marked differences in the characteristics of the people in the two districts, in the experimental district, the people tend to be younger, better educated, more urban, and somewhat better off economically than the people in the control district. Because of these differences, the people in the experimental district are probably more likely than the people in the control district to accept family planning, whether or not there is an incentive program. Once again, the question of validity arises. What is the true effect of a family planning incentive program? If both the experimental and control districts had been similar in the beginning, it might be possible to provide some answer to this question. But because there were initial differences, the effect of the incentive program cannot be distinguished separately from the difference due to a selection effect. 3 Testing Whenever a pretest is given, it tends to have an effect on the posttest. People who are given a pretest (trainees, for example) are likely to remember some of the questions and some of the errors they made when they take the pOsttest. They are likely to do somewhat better on the posttest than they did on the pretest. The difference or better performance on the posttest might have nothing to do with a program (such as a training course) but instead be due to the effect of the pretest. Sometimes in knowledge, attitude, and practice (KAP) surveys, the same group of people are repeatedly asked the same questions each year for several years. After awhile, they begin to remember the correct answer to many of the questions asked, such as, "Can you name three modem methods of contraception? Their responses may be due to their familiarity with the KAP survey and not to their increased knowledge of family planning as a result of an educational 31 program. Whenever a test is given repeatedly to the same group of people, there is every likelihood of a testing effect or threat to validity. 4 Instrumentation Whenever a measurement instrument (such as a questionnaire) is changed between the pretest and the posttest, this change is likely to result in an effect that is independent of any effect due to a program or project. For example, if knowledge of family planning is measured on the pretest by the question "Have you ever heard of family planning?" and then on the posttest by the question "Can you name three modem methods of contraception?" there will very likely be a difference in the percentage of people with "knowledge." The difference may be due to the change in the questionnaire instrument. Similarly, an instrumentation effect may be involved when interviewers become more experienced. An experienced interviewer may obtain more complete information from a respondent than an inexperienced interviewer. The additional information obtained may be due to the fact that the interviewer has become more skilled in asking questions or observing events, and not due to the effect of a program intervention. Changes in the way information is collected or changes in the instrument of data collection can result in a threat to validity. 5 Maturation Over time, people (and things) change, in long training programs, it is not unusual for trainees to become tired, hungry, bored, or discouraged. In longitudinal studies, particularly those that extend over a period of years, it is not unusual to find that respondents become more experi- enced, more knowledgeable, wiser, sometimes more resistant, and, of course, older. In other words, people mature over time. The maturation process can produce changes that are indepen- dent Of the changes a program intervention is designed to produce. If a posttest is given to a group of trainees who are tired, hungry, or bored, we might expect lower scores, on the average, than would occur if the trainees had not been tired, hungry, or bored. If these lower scores are then compared with the pretest scores, an evaluator might find there was very little difference and thus conclude that the training program was not successful. But is this really the case, or are the results just due to a maturation effect, another threat to validity? 6 Mortafit¥ In cohort studies (also called panel studies) where the same group of people are followed over time, there is almost always some dropout or loss of cases. It is often impossible to locate for a second survey all the same people who were interviewed in the first survey. Some cases will be lost to follow-up. Mortality effects refer to those losses. If the people who cannot be contacted are very different from those who can be contacted, there are likely to be great differences between the results of the first and second surveys. These differences may be due to the loss of cases rather than to the effect of a program intervention. Each of the threats to validity (history, selection, testing, instrumentation, maturation, and mortality) must be considered carefully when designing a research study. Unless controlled (accounted for), each validity threat represents a possible explanation for why a program pro- duced or did not produce an effect. In other words it is always necessary to distinguish an effect due to a program intervention fi_om an effect due to a validity threat. C Types of Study Designs In the following pages, several different research designs are presented and discussed. They range from nonexperimental to quasi-experimental to true experimental designs. The designs presented here are certainly not the only ones that exist, but they are some of the more frequently used designs in family planning operations research, in describing each of these, it is helpful to use a common notation developed by Campbell and Stanley.* This notation is explained in Figure 7.1. •See reference in biblio/.9"aphy. 32 FIGURE 7.1 Notation for study designs ExperimentalGroup _A'_ _" - RandomAssignmentof cases(subjects,respondents,patients,and so on)to the "_ ControlGroup experimentalgroupand the controlgroup. Time • - Thepassageof time. The extremeleft of a designisthe beginningof"a project or study.The extreme rightis the end. X - A programintervention,an experimentalintervention. O - An Observationmeasurement.The subscriptis usedto distinguishone observation measurementf,'om another.Forexample,01, 02, 03_04. O andX in a row - A row or line containinga singlegroupof subjects. - A separationof two groupsthat havenot been randomlyassigned. Exampleof a StudyDesign Time -_,_ ExperimentalGroup O1 X Oz _" ControlGroup O_ 04 In this design, all subjects are randomly assigned (RA) from a single population to the experimental group and the control group. Both the experimental and control groups receive an initial measurement observation (the pretests O1 and Oa). The experimental group then receives the program intervention (X), but the control group does not receive this intervention. Finally, a second set of measurement observations are made (Oz and 04). We would expect that, since the experimental group received a special program intervention (X), Oz would be greater than 04. Also, since both the experimental and control cases were randomly assigned, we would expect that O1 would be equal to O a on such key variables as age, sex, parity, and education. The design shown above is a true experimental design. Because it is one of the strongest in terms of controlling for threats to validity, it acts as a point of reference for the designs discussed below. 1 Nonexperimental Designs There are several nonexperimental designs commonly used by family planning researchers. These designs are most appropriate for collecting descriptive information or for doing small case studies of a particular situation. They are not recommended for evaluation studies that attempt to determine the effect of a program intervention, but they may be useful in diagnostic studies to determine the reasons why a problem exists. a Posttest-OnlyDesign Time P ExperimentalGroup X O_ In this design, a program intervention (X) has been introduced, and sometime after its introduc- tion, a measurement observation (O1) is made. Since there is no control group, there is no possibility of comparing the O 1 measurement with any other measurement. All that the O 1 measurement can do is provide descriptive information. The threats to validity of history, maturation, selection, and mortality are not controlled and therefore are factors to consider. This design would be most appropriate ff an investigator wanted to know, for example, the demo- graphic characteristics of IUD acceptors or sterilization acceptors. The X would represent the treatment (the IUD insertion or the sterilization operation). The O 1 would consist of information collected from a group of acceptors. While numerous problems are associated with the posttest- only design, particularly ff a comparative analysis is desired, much useful program information can be obtained if multivariate data analysis techniques are used (see Chapter 10, Section B.2.f). 33 b Pretest-Posttest Design Time ! Experimental Group O, X O, In this design, there is still no control group, but at least there is an earlier measurement observation (O,) that allows the research investigator to examine changes over time. But it is important to be aware that the pretest-posttest design is subject to several threats to validity-- history, testing, maturation, and instrumentation. c Static-Group Comparison Time P Experimental Group X O, Control Group Oa Unlike the other two designs, 'this one adds a control or comparison group. The experimental group receives a program intervention (X) followed by a measurement observation (O0. This measurement observation is then compared against a second observation (Oz) from a control group that did not receive the program intervention. Note, however, that the two groups are separated by a broken line (. ), which indicates that a random process was not used to create the two groups. This design might be employed if patients at one clinic were used as a control group to make comparisons with patients at another clinic where a special program intervention had taken place. The primary problem with this design is the threats to validity of selection and mortality. Initially, the two groups might differ greatly on the basis of such variables as age, parity, and education. Also, it is always possible that many of the patients in the experimental group who received the program intervention (X) may subsequently have been lost to follow-up for the O, • observation. Thus, the differences between 01 and Oz might be due to selection or mortality effects and not to the program intervention (X). 2 Experimental Designs a Pretest-PosttestControl Group Design Time ) [_ <_ Experimental Group O, X OzControl Group On 04 This design has already been presented earlier and discussed briefly. This is a true experimental design. The primary distinguishing feature is the random assignment of cases to the experimen- tal and the control group. Random assignment helps to assure that the two groups will be equal before the program intervention is introduced. In other words, if the randomization has been done properly, there will be a known probability of difference between the groups. Because the two groups are essentially equal at the beginning, any difference observed later on between Oz and 04 can be attributed tO the effect of the program intervention (X). This is one of the strongest designs in terms of controlling (or accounting for) threats to validity. At the same time, it is a design that is difficult to implement in field situations. Often it is not possible to assign cases randomly to experimental and control groups. For ethical or programmatic reasons, it is some- times impossible to deny program interventions to one group while offering them to another. There are, however, some situations where random assignment of cases can be done; for example, when the demand for a service is greater than the supply. This situation sometimes occurs when a new contraceptive method such as NORPLANT @ is introduced. The available supply of the contraceptive may be very limited but the demand potentially very high. A similar situation sometimes occurs when the number of persons who require training exceeds the number of places available in a particular training course. In these instances, experimental "cases" can be randomly selected from the pool of potential NORPLANT @clients or the pool of persons requiring training. Those persons not selected become the control group. b Posttest-OnlyControl Group Design Time [_] ._ Experimental Group X O, Control Group Oz This is also a true experimental design and similar to the one discussed above except there is no baseline (pretest) measurement observation. Since cases have been assigned randomly to the 34 experimental and the control group, these groups are assumed to be similar before the program intervention. This design allows the investigator to measure the effect of a program intervention on th e experimental group by comparing that group with the controls. But this design does not allow an investigator to determine the extent (or magnitude) of change within the experimental group because a. baseline pretest measurement was not taken. 3 Quasi-Experimental Design In many field research situations, it is simply impossible or certainly very costly and difficult to meet the random assignment criteria of a true experimental design. At the same time, researchers want to avoid the problems of threats to.validity associated with nonexperimental designs. A reasonable compromise often can be made by selecting a quasi-experimental design. These designs do not have the restrictions of random assignment. At the same time, they tend to control for mar_y threats to validity. a Time Series Design Time Experimental Group 01 02 03 X 04 Os 0_' The time series design is similar to the nonexperimental pretest-posttest design except that it has the advantage of repeated measurement observations before and after the program interven- tion (X). If, for example, we find that there is no difference between O_, 0 z, and O:_, but then a sudden increase/occurs between O 3 and O4, which is subsequently maintained in O._ and O_, we can conclude with some degree of confidence that the sudden increase was probably due to the effect 0f the program (X). This type of increase might be shown in graph form as in Figure 7.2. Suppose in this example the program intervention (X) is a large-scale educational cam- paign designed to increase people's knowledge and understanding of family planning. Before the educational program, observations O1, Oz, and O:_ reveal that only 20 percent of adults in a particular geographic area know and understand the meaning of family planning. After the program, observations O4,,O_, and O_ reveal that 40 pereent of the adults know and understand family planning. The most likely explanation for the change is the educational campaign. But another explanation might be a history effect or threat to validity that suddenly happened between Oa and 04. Although history and instrumentation effects are possible, the time series design helps the researcher avoid making mistaken or false conclusions. Study Figure 7.3. Note that there is a steady and constant percentage increase over time. The increase is no greater between O:_and 04, when the program intervention (X) took place, than it is at any other time. Obviously, in this situation the investigator could not claim that the program made a difference. This is the advantage of the time series design. The multiple measurement observations give us a trend. If the investigator had used the pretest-posttest design and compared only O:, against O,_, he or she might have mistakenly concluded that the program had made a difference. Now examine Figure 7.4. The trend in this example is regular and consistent increases and decreases in the percentage. The program intervention did not seem to make a difference in this trend. But, once again, if the investigator had used a pretest-posttest design and compared only O:_ against O 4, he or she might mistakenly have concluded that the program had had an impact. FIGURE 7.2 A sudden increase in a 60 measured variable after a program intervention r_ 40 /2O I I I I I I 01 O_ 03 t04 05 06 / Programintervention (X) 35 FIGURE 7.3 Steady increase in a _ 601 measured variable before and after a program intervention _ 4o 20 O, 02 Oj t04 05 061 Programintervention(X) FIGURE 7.4 Regular increases and 60 decreases in a measured variablebefore and after a program intervention _ 40 v *20 i I I I i I 01 02 03 104 Os 06 PrograminterventionO0 The time series design is a good one to use when you have access to regularly collected information, such as monthly service statistics. You can plot the information on a graph and then note the point at which a special program intervention occurred. If there is a sudden change in the plotted line that is unusual and not similar to the trend before or after a program intervention, you can be fairly sure that the sudden change was due to the program. Consider Figure 7.5. In this example, suppose that each month sterilization acceptors account for 30 percent of all new acceptors. Measurement observations O_, O z, and Oa all confirm the 30 percent figure. Then a major information and education campaign is conducted. Suddenly, sterilization acceptors account for 60 percent of all new acceptors, as shown through measure- ment observation 04. But soon after the campaign, sterilization acceptors drop back to the 30 percent level. The program intervention clearly had an impact, but the impact was not long lasting. Once again, had the investigator used a pretest-posttest design and compared only O a against 04, he or she would mistakenly have concluded that the information and education campaign had had an enormous impact. But the investigator would have missed the important point that the impact was only temporary. FIGURE 7.5 • 60f /_ Temporary impact of a program intervention *t20 01 Oz O_ t04 Os 061 Programintervention(X) 36 To summarize, although the time series design does not include a control group and does not control for history and possibly instrumentation threats to validity, it does allow for a more detailed analysis of data and program impact than the pretest-posttest design because it gives information on trends before and after a program intervention. It is a particularly appropriate design to use when it is possible for a researcher to make multiple measurement observations before and after a program intervention. b Nonequivalent Control Group Design TLme I Experimental Group . .O1._X.O.z. NonequivalentControl Group Oa 04 In field research settings, it usually is possible to compare an experimental group with a similar but not necessarily equivalent group. For example, suppose you want to study the effect of a • refresher training course given to family planning field-workers. Groups of 20 field-workers might be called into the training center for a three-week retraining program. These trair_ees could be given a pi'etest (O1), followed by the three-week training course (X) and then the posttest (Oz). For purposes of comparison, a similar group of field-workers who had not yet been given the refresher training course could be administered the same pretest at the same time (Oa) and the same posttest (04). In this situation, we could use the two pretests (01 and O a) to assess the extent to which the two groups of field-workers were truly similar. Then we would compare the two posttests (Oz and 04). We would expect that Oz would be greater than 04 because of the effect of the refresher training (X). The nonequivalent control group design is a particularly good one to use in evaluations of training programs. It is also a good design to use when a program intervention is introduced into one area (say, one district) and you want to compare the program effects in that district against a similar, but not necessarily equivalent, neighboring district. In using this design, you should give careful attention to the analysis and comparison of O 1 against O a (the pretest baselines). In particular, you should look for selection effects or major differences between the experimental and the nonequivalent control group that might help explain differences (or lack of differences) in the posttest O z and 04 comparisons. c Separate Sample Pretest-Posttest Design Time [_ PretestGroup O_ X ' [_ PosttestGroup X O2 This design is very frequently used in family planning research. Essentially, it involves doing a baseline pretest (01) with a randomly selected sample from a study population. Subsequently, a program intervention (X) is introduced, and then a posttest measurement (Oz) is made using a second randomly selected sample from the same study population. This design is a good one to use to avoid testing effects; however, it does not control for history, maturation, mortality, or possibly instrumentation effects. D Selectinga Study Design Selecting an appropriate operations research study design can be a tricky business. It usually involves a careful consideration of ethical issues and a balancing of technical issues against practical and administrative issues. 1 Ethical Issues. The first issues to consider in selecting a research design are the ethical ones. If a particular research design results in unethical procedures, a violation of people's rights and dignity, a denial of sewices that otherwise would be available, then the design should be modified or abandoned regardless of the effect this may have on reliability, validity, time, funds, available personnel. Indeed, if it is not possible to do an ethical study, then the study should not be done. There can be no compromising on this point. 2 Practical and Administrative Issues. Every researcher dreams of having available for a study lots of funds, time, personnel, and equipment (the latest computer, a nice laser jet printer, maybe a comfortable four-wheel drive vehicle). Unfortunately, dreams are not 37 reality. Most often funds are in short supply, time is short, personnel few in number, and equipment outdated or nonexistent. Obviously, these real-world conditions affect re- search and, in particular, the design of operations research. Available funds, time, and personnel may not permit three experimental groups and one control group. They may not permit a large and detailed baseline survey and another follow-up survey 24 months later. They may not permit running the study intervention for 36 months. All of these practical and administrative issues unavoidably affect the selection of a research design. 3 Technical Issues. While the technical objective in selecting a research design is to minimize possible errors, the dynamic field setting of most operations research studies usually presents numerous threats to the reliability and validity of the study. Almost always, experimental field studies face unanticipated events such as the closing of a service delivery center, the transfer of key personnel, a civil disturbance, a flood, or other happenings that disturb the research design or change the study intervention. The effects these events have on the reliability and validity of a study are often unclear or even unknown, but certainly they can be substantial. Some of the important technical issues you should keep in mind when selecting a research design are the following: a Whenever possible, try to create experimental and control groups by assigning cases randomly from a single population study group. b When random assignment is not possible, try to find a comparison grot/p that is as nearly equivalent to the experimental group as possible. c When neither a randomly assigned control group nor a similar comparison group is available, try to use a time series design that can provide information on trends before and after a program intervention (X). d If a time series design cannot be used, as a minimum and before a program starts, try to obtain baseline (pretest) information that can be compared against postprogram information (a pretest-posttest design). e If baseline (pretest) information is unavailable, be aware that you will be limited in the type of analysis you can conduct. You should consider using multivariate analytic techniques. f Always keep in mind the issue of validity. Are the measurements true? Do they do what they are supposed to do? Are there possible threats to validity (history, selecting, testing, maturation, mortality, or instrumentation) that might explain the results? E Guidelines for a "Good" ResearchDesign 1 A good research design is an ethical research design. It is a design that does not violate people's rights and dignity, and that does not deny people the level of quality of services that they otherwise would have. 2 A good research design is one that is capable of obtaining the most reliable and valid data possible given the constraints of funds, time, personnel, and equipment. 3 A good research design is one that is capable of measuring whatever it is that happens in a field setting, whether the impact of planned intervention activities or the impact of unplanned and possibly invalidating events. 4 A good research design is one that helps an investigator avoid making mistaken conclu- sions such as accepting a hypothesis when in fact the hypothesis is false, or rejecting a hypothesis when in fact the hypothesis is true. F The Principle of theThree Multiples As far as possible, it is always a good idea to keep in mind the principle of the three multiples whenever selecting a research design. 1 Seek multiple data sources to obtain information on the same variables. 2 Seek multiple measurements over time of the same variables. 3 Seek multiple replications of the study intervention in different field settings. 38 The use of multiple data sources serves several purposes. First, each source can provide a reliability check on the other sources. Second, each source may provide additional insights about a particular event or relationship between events. Third, the use of multiple data sources provides the opportunity to obtain qualitative information on process that can be particularly useful for determining how and why an intervention effect was obtained or not obtained. Multiple measurements over time of the same variables can provide information on trends before, during, and after the introduction of an intervention. This type of information can be extremely valuable for field studies. Sudden and radical deviations from past trendscan be the first indication that factors extraneous to a study intervention are affecting an experimental area or population. Multiple replications of a study intervention in different settings can provide information concerning the extent to which the intervention's effects are unique to a particular area and population, or can be generalized to other areas and populations. Ideally, the use of multiple replications means that one or more follow-up studies are conducted with the same objectives and research design, but are implemented in different areas. In practice, because of time and resource constraints, this is difficult to accomplish. An alternative is to introduce the intervention into several field settings at the same time. This procedure not only provides an indication concerning the degree to which the intevention's impact can be generalized, but also gives some guarantee that if one experimental area is affected by floods, riots, administrative delays, strikes, or other happenings, at least the study can continue in the other areas. What To Do---Designing a Study 1 The objectives of a research study determine to a large extent the type of research design required. Review the objectives of your study, and consider the following questions: a Are you planning to do an evaluation of a program? If your answer is yes, c.onsider using an experimental or a quasi-experimental design. b Are you interested only in describing a particular event? If the event has already taken place, you can use a posttest-only design (X O1) and consider multivariate analysis. If the event has not yet taken place, you might be able to use a pretest-posttest design (01X 02), a time series design (01 O 20 a X 0 40s 08), or a nonequivalent control group 01 x 02 05 04 e Will you be conducting an experiment to determine the effect of a particular approach on family planning service delivery? If your answer is yes, then you should seek to obtain a randomly assigned control group or at least a comparison group that is similar to the experimental group. 2 On the basis of your answers to the above questions, diagram with Os and Xs the type of design you think is most appropriate for your research study. Remember that many variations are possible to the designs presented earlier. 3 Consider the ethical implications, if any, of the design. 4 Be sure that available resources are adequate to implement the design. 5 Review the list of threats to validity and determine which ones are controlled and which ones are not. 39 Sampling A Selecting Area and Population In family planning operations research, the information that is collected refers most commonly to individuals. For instance, the effect of offering a new type of contraceptive may be studied by making it available to one group of eligible couples but not to another. Later, in postproject interviews, information can be obtained from the wives in both groups about their contraceptive practice. If the assignment of couples to experimental and control groups is randomized, such a design will permit a test of the hypothesis that availability of the new contraceptive method affects contraceptive practice. In such a study, the availability and use of the new contraceptive method are viewed as characteristics of the interviewed wives. Each wife may be referred to as an experimental subject. Since the wives are interviewed, they may also be called respondents. More generally, they may be referred to simply as eases, a useful term, since not all studies are experiments or surveys. Cases need not be individual persons; they may be aggregations or groups of persons. The relationship between the introduction of a new contraceptive and the practice of contraception described above could be studied at the community level. If the cases to be studied are entire villages (instead of individuals), the new contraceptive could be defined as "available" if it is offered by a clinic within a specified distance of the village; contraceptive practice could be measured in terms of the prevalence rate for the entire village (the percentage of married women of reproductive age using the method). Cases may also be organizational units. For instance, a study of clinics could be designed to determine the effect of offering a new contraceptive method on the method mix of new acceptors. Usually the type of case to be studied in a particular research project is clear from the nature of the project and its objectives. In a study to determine the effects of an intervention on knowledge and attitudes of target couples, it is clear that the appropriate cases to be studied will be target couples. In a study of the effects of training on field-workers' knowledge and skills, the appropriate cases for study are field-workers. In a study intended to compare high and low performing clinics, the appropriate cases are clinics. Some studies may ipvolve more than one type of case. A study to test the effects of a new training program for field-workers may be designed to permit observation of its effects on both the trainees and the eligible couples served by them. Such a design would call for analysis at two levels (at the trainee level and at the couple level), which would require separate data collection and processing for each level. B Selecting Cases Once the types of cases to be studied have been determined, the next step is to decide how the individual cases shall be selected. For this purpose, it is useful to distinguish between the total population and the sample from the population that will actually be observed. The population is composed of elements, each of which is a potential case. Some studies involve such small populations that all the elements can and should be included in the sample. For example, in a study of central management of a national program, there may.be a total of only 20 or 30 top managers available for the study. In this case, a 100 percent sample (the entire population of 40 managers) would be most desirable. Similarly, if there are only five mobile teams in a p!x)gram, a study of the operations of such teams should probably involve all five. Sometimes, 100 percent sampling is appropriate even for studies involving large populations. For instance, in a study of the effects of an intervention on the age distribution of acceptors, it may be possible to analyze from clinic records the data on all acceptors, especially if such data are computerized. Similarly, it may bepossible to use the entire population of clinics as the sample to study clinic performance. Usually, however, research problems involve so many persons or places or service points that it is necessary to select a sample of less than 100 percent to represent them. For a survey of family planning acceptors at program clinics during 1982, the population of concern would be the total number of persons registered as acceptors at program clinics during 1982. If this number is large---say, 800,000--it would be excessively costly, time-consuming, and unnecessary to inter- view them all. The usual practice is to select and interview a relatively small sample of acceptors--a representative subset of the total population. If the sample is truly representative of the population and consists of several hundred cases or more, the information obtained from it will be very similar to the information that would be obtained from a study of the entire population. There are two basic types of samples: (1) probability samples, and (2) non-probability samples. I Probability Sample A probability sample is more likely than a non-probability sample to be truly representative of the larger population from which it is drawn and is therefore usually considered preferable to a non- probability sample. The essence of probability sampling is that each element of the larger population (i.e., each couple, each field-worker, each clinic, etc.) has a known probability of being selected. (If each element has an equal probability of selection, the sample is known as a self-weighting sample, and the findings from analysis of the sample data can be taken as representative of the general population.) There are several ways of drawing probability samples, the most common being the following. a Simple Random Sampling In this sampling procedure, each element of the larger population is assigned a unique number, and a table of random numbers or a lottery technique is used to select elements, one at a time, until the desired sample size is reached. This procedure can be quite tedious in drawing large samples. b Systematic Sampling This is a modification of simple random sampling, which is ordinarily less time-consuming and easier to implement. The estimated number of elements in the larger population is divided by the desired sample size, yielding a sampling interval (let us call it n), and the sample is drawn by listing the population elements in an arbitrary order and selecting every nth case, starting with a randomly selected number between one and n. While simple random sampling requires a list of all population elements (called the sampling frame) in advance, systematic sampling permits concurrent listing and sampling. This can greatly reduce costs. Of course, it is preferable that the sampling frame already be available, since systematic sampling can then be used to obtain precisely the sample size desired. If the size of the population is not known in advance, the size of the sample also cannot be known in advance. For instance, if you want to draw a sample of 20 couples from a village, you must know the total number of couples in the village population. If that number is not known, it is necessary to estimate the number. If you estimate that there are 200 couples, the sampling interval should be 10, meaning that every tenth case is interviewed during the listing process. If your estimate is • too high, you will draw a smaller sample than you intended; if it is too low, your sample will be larger than intended. It is always better to have a larger sample than required instead of a smaller one than needed to do the research. Also, it is a good idea to budget for a somewhat larger sample than you need. 41 c Stratified Sampling Stratification may be used in conjunction with either simple random sampling or systematic sampling to ensure the desired representation of specific subgroups. For example, if it is espe- cially important that the age distribution of respondents in the sample be exactly the same as in the larger population, the list of elements in the larger population can be arranged by age categorymthat is, stratified by age--and then samples for each age stratum drawn separately. If the sample fraction is the same for all age strata, the age distribution will be the same in the sample as in the larger population. It is often useful, however, to apply different sampling fractions in different strata. For instance, in a study of family planning acceptors, it may be desirable to sample equal numbers of acceptors of pills, condoms, and sterilization, even though these methods account for different proportions of all acceptors. This could be accomplished by stratifying by contraceptive method and then applying different sampling fractions to the various strata. d Cluster Sampling Sampling can be simplified greatly be selecting clusters of elements (using simple random, systematic, or stratified sampling techniques) and then proceeding to study all elements in each of the sampled clusters. Most commonly the clusters are geographic units (e.g., provinces, districts, villages) or organizational units, such as clinics, Mothers' Clubs, or training groups. For instance, in order to obtain a sample of rural married women of reproductive age (MWRA), first clusters of rural villages could be sampled and then all MWRA interviewed in the sampled clusters. The chief drawbacks of this technique are: (1) it complicates the mathematics of calculating the sampling variance and therefore of testing for statistical significance; and (2) it usually requires larger sample sizes for achieving statistical significance at any given level. e Multistage Sampling In drawing samples from very large and diverse populations, the technique most commonly used is multistage sampling. The population is first divided into naturally occurring clusters, and then a sample of these clusters is drawn. This is called the first stage of sampling. The second stage consists of sampling individual elements (respondents, cases) from within each of the sampled clusters. Sometimes more than two stages are required. For example, provinces might be selected at the first stage, districts within the provinces at the second stage, villages within the districts at the third stage, and finally, individual respondents within the villages at the fourth stage. At each stage, simPle random, systematic, or stratified techniques might be used. f Probability Proportional to Size (PPS) This procedure is a variation on multistage sampling in which the probability of selecting a cluster is proportional to its size and an equal number of elements is sampled within each cluster. The PPS method of sampling is useful when the clusters vary greatly in size. In such situations, PPS often reduces both the sampling variance and the data collection costs. 2 Non-probability Sample Non-probability sampling (also known as convenience sampling) refers to the selection of a sample that is not based on known probabilities. Non-probability sampling may be either accidental (through the selection of whatever cases happen to be available), or it may be purposive (through the planned selection of specific types of cases). Non-probability sampling •procedures are not valid for obtaining a sample that is truly representative of a larger population group. Almost always, non-probability samples tend to overselect some population elements and underselect others. In the absence of known probabilities of selection, there is no precise way to adjust for such distortions. Despite these drawbacks, there are instances where trying to obtain a truly representative probability sample may be too difficult or too expensive. In such instances, non-probability smllpling pz_ovides the only viable means for collecting data. For instance, if a government ministry has been attempting to integrate family planning into the activities of its field-workers, it might be worthwhile to mount a study of the performance of such workers. If the research budget is too small to permit extensive travel to visit workers in many parts of the country, it may be 42 decided to sample field-workers in areas relatively nearby even though their experience may differ somewhat from that of the field-workers in more remote areas. Sometimes it is impossible or too expensive to find enough cases of a particular type by using probability sampling methods. For instance, if a study is intended to examine the effect of a particular radio program on family planning perceptions and attitudes of listeners, it would be costly to draw a probability sample of listeners. This would require, first, a complete enumeration of the broadcast area (or of a representative sample of subareas) to determine who has heard the program, and then, selection of a representative sample of listeners. If the funds available do not permit such a costly sampling scheme, the only alternative may be to go from house to house, interviewing listeners as they are found, until the desired sample size has been reached. The findings from such a sample would have to be treated with caution, since its composition would probably be affected by variations in ease of access. Rural listeners and slum dwellers, for example, might be underrepresented because they are relatively harder to find and interview than urban and more middle-class people. If non-probability sampling is necessary, efforts should be made to minimize bias. If any of the characteristics of the general population are known, the sample can be drawn in such a way as to ensure proportionate representation of the various categories of respondents. For example, if it is known that one-third of the population lives in urban areas and two-thirds in rural areas, the sample cases can be selected purposively from urban and rural areas in the same proportions. If the age distribution is known and is thought to be related to the key variables of the study, the sampling instructions might specify precise numbers to be interviewed in each age category. Such an approach to purposive sampling is called quota sampling, since quotas for specific subgroups are established at the outset. If the main purpose of a study is to maximize the variety of qualitative information from a group of respondents (for instance, to get a comprehensive view of how a particular strategy is implemented, to learn about a broad range of problems to be solved, or to get a wide variety of proposed solutions to existing problems), non-probability sampling may be the best approach. It permits the researcher deliberately to select a diversity of respondents and to seek persons who are most likely to provide articulate answers to complex questions. Typically, anthropologists use purposive samples when they obtain information about communities from selected key i_ormanta--persons who are knowledgeable, articulate, and drawn from the various subgroups in the community. C Sample Size Determination The size of the sample is determined primarily by two things: the availability of resources, which sets the upper limit of the sample size; and the requirements of the proposed plan of analysis, which sets the lower limit. For instance, the availability of trained interviewers to do a survey in a specified period of time may be insufficient to permit more than a certain number of interviews, in which case the sample size cannot exceed this number. On the other hand, the sample size must be large enough (1) to allow for reliable analysis of cross-tabulations; (2) to provide for desired levels of accuracy in estimates of proportions; and (3) to test for the significance of differences between proportions. If the resources available for a study are inadequate to obtain a sample of sufficient size, then the researcher must either find additional resources or revise the plans for data analysis. If you intend to analyze cross-tabulations of variables, you should keep two points in mind when determining sample size. First, each category of an independent variable included in a cross-tabulation should contain at least 50 cases, since percentages calculated on the basis of fewer than 50 cases tend to be unreliable. The minimum sample size required to ensure at least 50 cases in each category of the independent variable is obtained by dividing 50 by the proportion (P.) of total cases you expect in the smallest category of the variable: 50 m _ -- P_ The second point to keep in mind is that the expected number of cases in each cell of a table should be at least five. The sample required to ensure a minimum expected cell size of five is 43 obtained by dividing five by tile product of tile proportions falling in the smallest categories of the variables in the cross-tabulation table. To illustrate these two points, consider the sample size required to analyze a cross- tabulation of contraceptive practice and educational attainment shown in Table 8.1. Education would be the independent variable in Table 8.1. Hence, in order to have a sample large enough to ensure that at least 50 cases will fall into the smallest catego(y of the variable educational attainment (that is, into the category "college," which accounts for only 5 percent of the total number of cases), we need to divide 50 by 5 percent (n = 50/.05 = 1,000 cases required). To find the minimum sample size needed to ensure an expected cell frequency of at least 5, we must divide 5 by the product of the proportions falling in the smallest categories of the two variables (.05 for the categoly "college" and .20 for the categoi_y "past user"): 5 n = _ = 500 cases required(.05)(20) Since the sample size required must meet both criteria (50 cases in each variable category and 5 expected cases in each cell), the larger estimate of n = 1,000 should be selected instead of the smaller estimate of n = 500. To calculate the minimum sample size required for accuracy in estimating proportions, you need first to answer the following questions: t What are reasonable estimates of key proportions to be measured in the study? For example, if you are studying contraceptive prevalence, you should try to guess what prevalence rate you will obtain. If you cannot guess what it will be, the safest procedure is to assume it is .50 (50 percent), which maximizes the expected variance and therefore indicates a sample size that is sure to be large enough. 2 What degree of accuracy do you want to have in your study? How far can you allow the sample estimates of key proportions to deviate from the true proportions in the popula- tion as a whole? For instance, if you find that the sample estimate of the prevalence rate is .50, do you want to be confident thatthis finding is accurate within 1 percent or 5 percent (usually referred to as the .01 and .05 level, respectively)? If you seek a high degree of accuracy (such as .01), your sample will need to be much larger than if you seek a relatively low degree of accuracy (such as .05). 3 What confidence level do you want to use? How confident do you want to be that the sample estimate is as accurate as you wish? Customarily, the 95 percent confidence level is specified. 4 What is the size of the population that the sample is supposed to represent? If it is greater than 10,000, the precise magnitude is not likely to be very important. But if it is less than 10,000, the required sample size may be smaller. 5 If you are seeking to measure the difference between the two subgroups with regard to a proportion, what is the minimum difference you expect to find statistically significant? For instance, if you are comparing the contraceptive prevalence of an experimental group and a control group, and you find a difference of only 5 percentage points, do you expect a difference this small to be statistically significant? The smaller the difference you expect to be significant, the larger your subsample sizes will have to be. TABLE 8.1 Cross-tabulation of contraceptive practice and educational attainment Educational attainment Secondary Primary No school school schooling Contraceptive CoHere (15 per-" (40 per- (40 per- practice (5 percent) cent) cent) cent) _UlTentuser 30percent) 'ast user _0percent} _ever user _,0percent) 44 On the basis of your answers to these five questions, you can calculate the sample size needed to measure a given proposition with a given degree of accuracy at a given level of statistical significance by using a simple formula, provided that the total population size is greater than 10,000: z=pq ri -_ --d= _,,Vhere • n = the desired sample size (when population is greater than 10,000). z = the standard normal deviate, usually set at 1.96 (or more simply at 2.0), which corresponds to the 95 percent confidence level. p = the proportion in the target population estimated to have a particular characteristic. If there is no reasonable estimate, then use 50 percent (.50). q= 1.0 - p. d = degree of accuracy desired, usually set at .05 or occasionally at .02. For example, if the proportion of a target population with a certain characteristic is .50, the z statistic is 1.96, and we desire accuracy at the .05 level, then the sample size is: (1.96)z(50)(50) rl---- (.05)= = 384 If we use the more convenient 2.0 for the z statistic, then the sample size is: (2)=(.5o)(.5o) rl - -- (.05)z = 400 Note that the numerator in this case is 1.0. This means that when you assume the proportion is _0 and set a 95 percent confidence level by using z equal to 2.0, then the formula for sample size is simply: 1.0 d2 IfN (the entire population) is less than 10,000, the required sample size will be smaller. In such cases, calculate a final sample estimate (n_ by using the following formula: rl ns= Where: nf = the desired sample size (when population is less than 10,000). n = the desired sample size (when the population is more than 10,000). N = the estimate of the population size. For example, if n were found to be 400 and if the population size were estimated at 1,000, then n£ would be calculated as follows: 400 nf= 40o 1.4 -- 286 If you wish to test differences (d') between two subsamples regarding a proportion and can assume an equal number of cases (nl = nz = n) in the two subsamples, the formula forn' is very similar to the one given above: rl f = 2z___.q(d')= For example, suppose you want to compare an experimental group against a controlgroup with regard to the proportion of persons using contraception. If you expect p (the proportion 45 using) to be .40 and you wish to conclude that an observed difference of .10 or more is significant at the .05 level, then the size of the sample can be calculated as follows: 2 (1.96)z(.40)(.60) n' -- -- 184 (.lOP Thus, we need 184 experimental subjects and another 184 control subjects. Note that, as with the earlier example, the formula can be simplified to the following ff p is set at .50 and z at 2.0: 2 (d')Z If this formula were applied to the example above, the sample size would be calculated as: 2 (.lOP = 200 If you cannot assume equal subsample sizes or if the sizes of the populations from which the subsamples are drawn are much less than 10,000, the procedure for estimating sample sizes is more complicated. In such cases, or in any others where you are confused by sampling, you should consult a statistician. What To Do--Sampling I Decide first whether you want to draw a sample and, if so, whether it should be a probability sample or a non-probability sample. In making this decision, take into account the objectives of the study, the extent to which you want the findings to be representative of a larger population, and such resource factors as cost, time, and personnel. Calculate the size of the samplerequiredforyour study.You may use the formulas given above, but you should also seek the assistance of a statistician if possible. The statistician will need to know your estimates of the proportions to be tested, the degree of accuracy you seek or magnitudes of differences you wish to test, the confidence levels you wish to use, and the approximate size of the population from which the sample will be drawn. 3 Bear in mind the following basic principles: a A larger sample will yield more accurate results but will be more cosily than a smaller sample. b A probability sample will provide quantitative data more representative of a larger popula- tion than will a non-probability sample, but a non-probability sample can be designed in such a way as to maximize insightful qualitative data from relatively small samples. e If your proposed analysis calls for studying particular subgroups of your sample, your sample size will need to be expanded accordingly. For example, to study characteristics of a group of acceptors, you may need a sample of only 400 acceptors, but if you want to extend the analysis to the acceptors of particular methods, the subsample sizes will be too small to yield significant findings unless the total sample size is increased. Even ff statistical signifi- cance is not considered very important, there should be at least 50 cases in the smallest subgroup to be studied if you want to obtain even moderately reliable percentages. 46 Data Collection There are many different ways to collect data. The approach selected depends on the study objectives, the study design, and the availability of time, money, and personnel. An important consideration in deciding on the best way to collect data is whether the study is intended to produce relatively precise quantitative findings or to produce qualitative descriptive informa- tion. Most operations research studies are concerned with the quantitative measurement of program operations, but many also are (or should be) concerned with detailed qualitative information on processes (e.g., how a project is actually implemented in the field or how couples decide to use one contraceptive method in preference to the others available) or perceptions (e.g., potential acceptors' understanding of how a method is used and its possible effects or perceived problems with existing program se[vices). Often study objectives call for both quantitative and qualitative information, which may imply that more than one data collection method should be used. A Quantitative Data The most common means for collecting quantitative data on people is the structured interview. However, such information can sometimes be collected in other ways: from self-administered questionnaires, service statistics, or such secondary sources as the census, vital records, and other existing records and reports. If the sample is composed of geographic or organizational units (such as villages, districts, clinics, hospitals) rather than people, quantitative data usually can be obtained from service statistics and secondary sources. If information is not available from such sources, it may be necessary to obtain data on geographic or organizational units by interviewing people who are members of these units or knowledgeable about them. I Structured Interviews Studies that obtain data by interviewing people are called surveys. If the people interviewed are a representative sample of a larger population, such studies are called sample surveys. If the sample is large enough to permit statistical analysis, it is customary to employ structured interviews rather than unstructured ones, since the former lend themselves better to quantitative analysis and the latter create serious data processing difficulties, particularly if the sample is large. A structured interview is one that employs a standard questionnaire (or interview schedule) to ensure that all respondents are asked exactly the same set of questions in the same sequence. The exact wording of each question is specified in advance, and the interviewer merely reads each question to the respondent. In designing a questionnaire and then using interviewers to administer it, there are several points to remember: a Use simple language that x_dll be understood easily by the respondents. b Whenever possible, precode the responses to the questions so that the information can be transferred easily to computer cards or otherwise tabulated. This requires more effort when designing theintetview schedule, but the savings of time and effort during the processing and analysis more than compensate for it. 47 c Try to avoid embarrassing or painful questions. If it is necessary to ask a sensitive question, word the question as tactfully as possible and avoid asking it near the beginning of the interview, when the respondent may not yet be completely relaxed. d Avoid asking for more than one item of information in a single question. For instance, do not ask "Do you and your husband want another child?" If the respondent and her husband disagree about having another child, an answer of either "yes" or "no" will be impossible to interpret accurately. A response of "yes" could mean "l want another child," "My husband wants a child," or "Both of us want a child." It would be better to ask two separate questions: "Do you want another child? . Does your hus- band want another child?" e Watch out for ambiguous wording of questions. For instance, if family planning has already been defined, a question such as "Which method or methods of family plan- ning do you use?" may seem clear enough. But if the respondents are wives, they may not mention use of vasectomy or condoms, since these are male methods. The question should be "Which method or methods of family planning do you and your husband use?" f Do not overload your interview schedule with items that are not essential for your study. Keep it as short as possible to avoid tiring your respondent and to simplify the data processing and analysis. g On the other hand, be sure to include all questions necessary to provide sufficient information on the variables you want to study. Also, be sure that the data necessary to test the hypotheses of the study can be obtained from the data collection instru- ments. It is often helpful to prepare a list of key study variables with an indication of where the data for the variable will be obtained. h Start with the easier questions, moving on to the ones that are more sensitive or difficult to answer only after the respondent has had an opportunity to become accustomed to the interview situation. Respondents are likely to be somewhat tense or even suspicious at first, and a major task of the interviewer during the first few minutes is to establish rapport--to place the respondent at ease. This task is facilitated ff the initial questions are not likely to cause embarrassment or be difficult to answer. i Ask all respondents each question in exactly the same way. If the interview is to be conducted in more than one language, prepare full written translations into all major languages and instruct your interviewers to use those translations word for word. Do not permit free translations, except for languages with too few respondents to justify the cost of preparing written translations. To ensure comparability of wording among the various written translations, have them back-translated into the original language to verify that the meaning is retained. The back translation should be done by persons who are not familiar with the original wording of the questionnaire. j Pretest the questionnaire in an actual field situation. There are several principles of pretesting that should be noted: 1 The pretest need not involve large numbers of respondents; 30-50 is often enough if the respondents are sampled (purposively) in such a way as to ensure that the expected heterogeneity of the study sample is reflected in the pretest sample. This means making sure that the pretest includes old and young, urban and rural, less educated and more educated, contraceptors and noncontraceptors, and so on. 2 Be prepared to conduct more than one pretest. If a pretest results in major revisions, it is a good idea to conduct a second pretest to be sure the revisions are satisfactory. 3 Pretesting should be completed before the training of interviewers. Often it is possible to use field supervisors to do the pretest. This gives them a clearer understanding of study objectives and better prepares them to assist in the training of interviewers. 4 The main purpose of the pretes t is to ensure that the respondents are able to understand the questions and answer them usefully. Hence, it is not enough simply to interview the pretest respondents; each interview should be followed by a debriefing, in which the interviewer asks about the respondent's under- 48 standing of questions that are thought likely to be misunderstood or that appear to have caused difficulty during the interview. k IhPovide complete training for all interviewers. The training should be designed to familiarize the interviewers with the intent and meaning of the questions, let them role play interview situations, and give them experience in actually conducting interviews in the field under supervision. l If a prospective respondent is not available during the interviewer's first visit, an appointment should be made for a call-back visit. It is common to require at least two caU-backs before dismissing a sample case as unavailable for interview. m If the sample is small, it may be necessary to find substitutes for cases that cannot be located. If so, the interviewers must be given clear instructions for obtaining substitutes that will ensure either a random sampling of substitutes or selection of substitutes who are similar to the cases originally selected. However, if the sample is large enough to tolerate some loss of cases, it is usually better not to draw substitutes. n Every effort should be made to isolate the respondent during the inte_iew. If others are present, the respondent's answers may be influenced. For example, a respondent whose husband does not know she is using oral contraceptives may give the inter- viewer one type of answer to a question if she is interviewed alone, but quite another type of answer if her husband is present. o Cheek all completed interview schedules for errors, omissions, and discrepancies as soon after interviewing as possible. Respondents should be revisited to correct errors that cannot be otherwise resolved. It is best if the interviewer can check the question° naire immediately after the interview so that the respondent can be consulted. After the interviewer has checked and corrected the questionnaire, it should be rechecked by the field supervisor. This checking process is known as field editing. 2 Service Statistics All national family planning organizations generate seDAce statistics. Some organizations have established a management information system (MIS). The quality of service statistics, however, varies from country to country and even within countries. Thus, they should be used with caution. Service statistics often help a researcher define the parameters of the problem he or she wants to study. In some cases, they can be used to compare the results of a particular study with nationwide figures. Often in operations research projects it is necessary to design supplementary forms to provide data that are not available from the regular service statistics. 3 Self-Administered Questionnaires If money, personnel, and time allow, interviews are generally preferable to self-administered questionnaires. For most surveys in developing countries, self-administered questionnaires are difficult if not impossible since many respondents are not well enough educated to complete questionnaires themselves. There are other problems with self-administered questionnaires, even among educated respondents: Instructions or questions are more likely to be misunderstood without an interviewer to help explain them; portions of the questionnaire are more likely to be left blank; it is difficult to incorporate many conditional sequences of questions (e.g., "If the response to question 12 is 'yes,' go to question 13; if not, skip to question 18") without causing confusion. Self-administered questionnaires are likely to be useful in situations where literate respon- dents are already gathered together in a setting where they can write--for example, in a classroom or an office. Self-administered questionnaires can be especially useful in evaluating population education programs or training programs. Sometimes blank questionnaires are mailed out to respondents, who are requested to complete them and send them back. This has the virtue of being very inexpensive, but it has all the drawbacks noted above, plus the added problem of high nonresponse rates. It is common for mailed questionnaires to elicit less than a 50 percent return, even after one or two reminders. This drawback affects the representativeness of the sample and may render the validity of quantitative findings so questionable that they are of very little use. 49 4 Secondary Data Sources Information fl'om recent censuses, vital records, other official statistics, and even previous surveys often call be incorporated with data collected specially for a study to enrich the analysis. A large body of data on health and family planning has been collected by worldwide surveys such as the World Fertility Surveys, the Contraceptive Prevalence Surveys, and the more recent Demographic and Health Surveys. B Qualitative Data The data collection techniques most appropriate for studies whose objectives call for descriptive, qualitative data tend to be different from those most appropriate for quantitative analysis. Most operations research studies should use a combination of quantitative and qualitative data collec- tion methods in order to obtain the most accurate and realistic picture of a program situation. Quantitative methods discussed earlier are important to obtain data for making predictions, probabilistic statements, and generalizations. Qualitative methods such as unstructured inter- views, focus group discussions, direct observation of operations, and content analysis are impor- tant to obtain data on processes, on how and why a program works, and on unintended and unanticipated program outcomes. 1 Unstructured Interviews The chief drawback of structured interviews is that the responses obtained tend to be superficial. An alternative approach to interviewing, which permits greater depth of meaning, is to seek detailed, open-ended responses to questions. Such interviews are often called in-depth inter- views. Instead of reading formal questions from a structured interview schedule, the interviewer has an outline of topics or a set of general questions to serve as a guide to the kind of information required. Details that are not brought out initially are sought through follow-up questions, called probes. The chief drawbacks of unstructured interviews are (1) the responses, being unstandard- ized, are difficult to quantify; (2) the interviews require highly skilled and experienced inter- viewers; and (3) the analysis is exceedingly time-consuming. The shortage of qualified inter- viewers and analysts, and the high cost of conducting and processing such interviews usually mean a small sample size must be used (sometimes as few as 20-30 respondents). In-depth interviews are usually most useful exploratory studies that seek to clarify concepts or generate hypotheses prior to developing questionnaires for quantitative surveys. They also are useful for generating supplementary, explanatory data to augment the relatively superficial findings from larger surveys. For example, little is known about the motivations of men who seek vasectomy. An unstructured interview could assist in gaining insight into the dynamics and determinants of the decision to adopt vasectomy. 2 Focus Group Discussion A way of reducing the amount of time and personnel required for conducting and analyzing in- depth interviews and yet getting detailed qualitative information from a relatively large number of respondents is to bring the respondents together in groups. The interviewer (or facilitator) follows much the same procedure as in the unstructured interview situation, using a general discussion guide and eliciting details through probes. Participants are usually sampled purposively to reflect population variations that are of particular relevance to the research topic. For instance, cases sampled for focus group discussions on natural family planning (NFP) might involve purposively selected samples of husbands, wives, current users of each type of NFP (calendar rhythm, temperature method, cervical mucus method, and sympto-thennal method), past users no longer using NFP, and never users. 3 Direct Observation of Operations This technique can generate either quantitative or qualitative data but tends to be used move for small-scale exploratory studies than for large-scale quantitative studies. This is because it usually 5O requires relatively highly skilled observers and analysts and prolonged periods of observation, resulting in a high cost per unit of observation. This technique lends itself particularly well to observation of community responses to program efforts. It is the chief method of ethnographers, who specialize in community studies. It is also useful for organizational studies, such as observa- tion of clinic operations, activities of field-workers, and administrative procedures. 4 Content Analysis of Written Materials This is usually not employed as the sole method in a family planning operations research study, but it can serve as a useful adjunct to other types of data collection and analysis. For instance, documents related to a training curriculum may be content analyzed to determine just what knowledge and skills the training is supposed to develop. The content analysis can serve to guide the researcher in devising procedures to test the knowledge and skills of the trainees. IEC materials may be content analyzed to indicate whether messages are being overemphasized or underemphasized in relation to program needs. Research reports may be content analyzed to determine the present state of knowledge about a particular research topic so as to guide future research efforts. The content of press reports or public statements made by policymakers may be studied to assess attitudes towards family planning and population issues. What To Do--Data Collection 1 Review your study objectives and hypotheses as well as your list of independent and depen- dent variables. What types of information do you require? Which data gathering technique(s) would be most appropriate and feasible for gathering the desired information? Is some of the information already available from other sources? 2 If you intend to collect information through a survey, review the list of points under the heading Structured Interviews. Be sure your proposal discusses important steps, such as translation, pretesting, training of interviewers, and rules regarding call-backs and substitu- tions. 3 Make an outline of the data gathering instrument(s) (e.g., interview schedule, discussion guide, observation guide) that you intend to use. Give examples in the proposal of questions to be asked, especially those designed to elicit information regarding key variables found in the hypotheses. Be sure you incorporate measures (or at least descriptions of measures) of all the variables you intend to study. It is useful to list all the variables of the study and then under each variable record the question or questions that will be used in the questionnaire. For example: Variable l" Family planning practice QI: Have you ever used any method of family planning? Q2: Are you currently using any method of family planning? Variable 2: Education QI: What is the highest class in school you have completed? 4 Prepare a description of your data collection procedures and include it with the outline of the data gathering instrument(s) in the proposal. C Data Quality Checks There are several ways to check the quality of interview data: 1 Sometimes a researcher will deliberately ask two or more questions that give the same type of information. The first question might be asked at the beginning of the interview and the second at the end. The two questions are then examined for consistency of response. This is one way to check the reliability of the data. 2 For difficult questions, sensitive questions, or questions where the researchers want to be sure the information is correct, the interviewers can be instructed to probe. That is to 51 say, the interviewer can repeal tile question in a slightly different form or repeat the respondent's answer and then ask if the information is correct. For example, a woman might report that she has two sons and three daughters. The inte]viewer might then say, "You have a total of five children, two male and three female. Is that correct? Are there any other children whona you may have forgotten to tell me about?" 3 Field supervisors should be used to help the interviewers with difficult situations and to check that they are actually doing their work. Occasionally, interviewers complete their questionnaires in lea stalls! Some studies use a ratio of one supervisor for every five interviewers. 4 Most studies that use an interview procedure make an attempt to reinterview a certain percentage of the respondents. Depending upon the size of the sample, a general rule is to reinte_view between 5 and 10 percent of the sample. The data fi'om the first interview are then checked against the data fi'om the second interview for consistency. This is another check on the reliability of the data. Obviously, if there are major inconsistencies, particularly on such questions as age, marital status, and parity, then clearly there is a problem somewhere. Tile problem might be with the questionnaire, with the inter- viewers, with the tabulation procedures, or in some other area. Once the data have been collected and tabulated, it is possible to do statistical checks for errors or for consistency of response. For example, a frequency distribution of the parity of women.may reveal that several women claim to have 18 or 19 living children. Since this is highly unlikely, the investigator is faced with the choice of either discarding these questionnaires, eliminating the information from the question on parity, or going back and reinterviewing the women Who claim to have 18 or 19 children. What To DomData Checks 1 Describe the procedures you will use to check the quality of the data collected. Consider the following procedures: a Include repeat questions in your questionnaire that can be used to check for consistency of response. b Have supervisors monitor the work of the interviewers in the field. c Reinterview a percentage of respondents, and look for inconsistencies. d Recode a percentage of the questionnaires to be sure that there are no coding errors. e Examine the frequency distribution on all variables to see if there are odd codes or items that are not logical. D Confidentiafityoflnformation It is always important to maintain the confidentiality of the information collected from respon- dents. Unless absolutely required, do not collect information that is sensitive. Whenever possible, use code numbers instead of names. Assure the respondent that the information he or she gives will be kept confidential. Do not let other people use the information you have collected when there is a chance that the use of the information could be damaging to the respondent. You have an obligation to protect the confidentiality of your respondents. Also, if a respondent does not want to be interviewed, you have an obligation to respect his or her wishes. Use an Informed Consent Form to explain the basic nature of the study and seek the agreement of the respondent to be interviewed. What To DomConfidentiality of Information I Describe in detail how you plan to maintain the confidentiality of infolxnation collected. 2 Include a sample of the Informed Consent Form in your study proposal. Tabulation and Analysisof Data If the only purpose of your proposed study were to present qualitative, descriptive findings, analysis would consist simply of rearranging the raw information according to an outline and then writing a narrative report incorporating it. Most studies, however, especially in operations research, involve quantitative manipulation of the information collected, in which case the analysis phase is more complex. First, you need to decide whether the tabulation of data will be performed by hand or by machine. Second, you need to decide how the information can be converted into a form that will allow it to be processed efficiently. Third, you must specify the statistical manipulations to be performed. Finally, you need to present the important findings resulting from these manipulations in a report or series of reports. A Preparing Tabulations Microcomputers can be found throughout the world in universities, research institutes, minis- tries of health, and family planning organizations. They have become an essential tool for the tabulation and analysis of operations research data. Even in small studies that employ relatively simple statistical procedures, computer tabulation of data is a good idea. Micro and laptop computers can produce tables and perform statistical operations much more easily (and usually with far less error) than is possible by hand. Once you have decided how tabulations will be generated, you must consider how the raw information will be transformed for tabulation and analysis. Regardless of whether tabulation will be done by hand or by computer, non-numerical data that are to be analyzed quantitatively must be converted into numerical codes. In addition, they usually must be transferred from their raw form (marks on interview schedules or questionnaires, transcripts of interviews, published census reports, etc.) to a form that is better suited for tabulation. For hand tabulation, it is often useful to transfer coded data to cards specially designed for hand sorting. For computer input, the information should be coded on code sheets and then entered on tape, hard disks, or diskettes. Data coding instructions must be prepared. For all variables studied, the coding instruc- tions convert each of the possible response categories into a unique numerical code. If the number of categories for a particular variable (including, if relevant, codes for "noninformation," "not applicable," "don't know," and "others") is fewer than 10, numerical codes should be single- digit numerals. If the total possible number of categories is between 10 and 99, two-digit codes should be used instead. For some variables, it may be necessary to use three-digit, four-digit, or even larger code numbers. Coded information should be spot-checked for errors. Some proportion of the coding should be repeated independently by a person other than the original coder and the results compared. If discrepancies are found, they should be checked and corrected by the coding supervisor. Most often, discrepancies will be due to problems with the coding instructions or poor training of the coders. After the information has all been coded, it should be edited (even if the raw data have already been edited in the field). The coded data can be edited primarily by searching for the following types of errors: 53 1 "Illegal" Codes--values that are not specified in the coding instructions. 2 Omissions--for example, a failure by an interviewer to follow correctly the SKIP instruc- tions tin a questionnaire. 3 Logical Inconsistencies---for example, a respondent whose current age is less than her age at marriage. 4 Improbabilities--for example, a 25-year-old woman with 10 living children. Once found, errors should be checked by reference to the original data forms and the necessary colrections made. Editing of coded data is most easily performed on the computer and is referred to as machine editing to distinguish it from field editing. Generally, data for computer processing are kept on tapes, hard disks, or diskettes. What To DomCoding and Editing 1 Decide whether the data should be tabulated by hand or computer. Take into account: a Availability of facilities. b Nature of the data (quantitative or qualitative). c Type of analysis to be performed. d Personnel available. e Costs. If you decide to use a computer, be sure to check the availability of existing (package) programs and programming assistance. 2 Decide which information is to be coded. Some information may be best left uncoded. Even a structured interview schedule may contain some open-ended items that lend themselves best to descriptive analysis. If so, indicate which items will be analyzed quantitatively. 3 Indicate in the proposal what provisions will be made for spot-checking and editing the coded data. B Plan for Data Analysis 1 Attributes of the Data One of the. most important parts of a research study proposal is the plan for data analysis. The purpose of data analysis is to provide answers to the research questions being studied. The analysis breaks the data into parts. The data are categorized, arranged, and summarized. Usually, a researcher wants to find from the data one or more of the following attributes: a Central Tendency or Characteristic of the Data Suppose a researcher wants to know the mean parity of a group of women, the most frequently used method of contraception (the mode), or the median level of the income. The mean, the mode, and the median are measures of central tendency. The mean is simply an arithmetic average. It is the sum of individual scores or measures divided by the number of individuals. The mode is the most frequently occurring figure in a set of figures. The median is the midpoint measure of a group of measures. In a slightly different' way, each of these statistical measures describes the typical characteristic or tendency of a group. b Variance in the Data Often a researcher is interested not only in the average characteristic of a group but also in the variation within the group. The researcher may want to know how individuals within a group differ from the average or central tendency of the group. For example, ff the average age of a gtx)up of pill acceptors is 28, what is the range in ages of the group? What is the age of the youngest pill acceptor and the oldest? A common statistical measure of variation within a group is the standard deviation. This measure gives the average distance of individual measurement observa- tions from the group mean. 54 c Differences within the Data Most often researchers are interested in knowing whether the findings they observe from their data could have occurred by chance alone. For example, if 43 percent of an experimental group of 1,000 married women are currently using a method of contraception compared with only 21 percent of the control group, is this difference statistically significant or could it have occurred by chance alone? Probably the most commonly used statistical tests for differences are the chi- square (X2) test and the t test for the difference between means. The use of these test measures is discussed later. d Relationships within the Data Most studies try to determine whether a relationship exists between one variable and another. For example, is contraceptive acceptance related to age? Is improved work performance related to training? Are radio family planning messages related to increased family planning knowledge? Is the level of education of a person related to contraceptive use? There are many different statistical measures of relationship, such as the product-moment correlation or the nonparametric correla- tion coefficient gamma. These and other measures are discussed later in this section. 2 Types of Analytical Procedures In a proposal, the plan of analysis will probably describe a number of different analytical procedures the researcher plans to use to determine the central tendencies of the data, the variance, the differences, and the relationships among variables. Among the most commonly used types of quantitative analytical procedures in family planning operations research are: a Variable transformations b Univariate analysis c Time series analysis d Comparisons e Bivariate analysis f Multivariate analysis g Cost-effectiveness analysis h Use-effectiveness analysis i Fertility analysis You need to indicate in your proposal which of these analytical approaches are most appropriate for meeting your study objectives. A prime consideration in making a selection is the extent to which the analytical procedure is appropriate to answer your research questions. Another important factor to consider is the extent to which you or someone associated with the data processing and analysis understands the terms and operations involved in the analytical procedure chosen and can interpret the results correctly. The purpose of the following discussion is merely to indicate the options for data analysis available to you. The discussion cannot teach you how to analyze data if you are not already . familiar with analytical procedures. All statistical tests and measures referred to below are widely used by social scientists. The reader who wishes to learn how to calculate and/or interpret them can find discussions and examples in almost any introductory textbook on statistical methods. a Variable Transformations It is often necessary to transform variables for subsequel_It analysis. For instance, instead of the respondent's age, the inte_iew questionnaire may record only the month and year of birth. If age is a variable to be studied, it can be obtained simply by subtracting the month and year of birth from the month and year of interview. Since this calculation can be done more easily and accurately by the computer than by hand, it should be done during the analysis phase of the research rather than during coding. This transformed variable might be transformed further still for certain kinds Of additional analysis. For instance, if you wish to cross-tabulate age by other variables, it is preferable to limit the age distribution to relatively few age categories (usually 5- or 10-year categories) or even to dichotomize (e.g., ages 15-29 and ages 30 or more). There are several ways to transform variables; among the most common are the following. Recodes. in recoding, category labels are changed. This technique is used for collapsing large numbers of variable categories into smaller numbers. For example, single years of age can be collapsed and transformed into age categories, such as 15-19, 20-24, 25-29. Counts. if you are collecting information on whether the respondents have ever used each of 10 family planning methods, you might want to count the number of methods ever used by each respondent, thereby generating a new variable that might be called "Number of Methods Ever Used." Counting is useful in studies involving more than one variable. Scaling. A variety of scaling techniques are used in family planning operations research studies. Probably the most common is the Likert scaling technique, sometimes used for studying atti- tudes. Likert scales are constructed by presenting respondents with a series of statements that reflect different aspects of an underlying attitude in a variety of ways (e.g., pills cause cancer; pills are highly effective; pills are simple to use; pills cause discomfort). The respondent is then asked about each statement whether he or she agrees strongly, agrees moderately, or disagrees strongly. Code numbers 1-5 are assigned to each response, a code of 1 indicating a strongly favorable attitude and 5 a strongly unfavorable attitude (or vice versa). The scores for all attitude statements are added and then divided by the number of statements, yielding a scale value between 1.0 and 5.0. Conditional Transformations. Where the nature of the transformation of one.variable depends on the second variable, conditional transformations may be useful. For instance, suppose you have asked respondents three questions: (1) How many children do you have? (2) Do you want more? (3) If yes, how many more do you want? With the information from these three questions, you can then create a new variable called "Total Desired Number Of Children." This can be done by adding the existing number of children (from question 1) to the additional number desired (from question 3). The number in quesiion 3 would be added to the number from question 1 only if the answer to question 2 is "yes." Other Mathematical Transformations. The calculation of age from the date of birth and the date of interview is an example of a mathematical transformation. Another example, from community- level analysis, is to divide the number of contraceptive users in a community by the number of eligible couples to obtain a community prevalence rate. b Univariate Analysis As the name implies, this form of analysis examines single variables. Often it is important to analyze the distribution of sample cases with regard to a single variable. For example, the age distribution of a sample population might be studied to be sure it is consistent with the age distribution of the larger population from which the sample is drawn. Univariate analysis of sample data can be used to indicate the composition of the larger population with regard to important study variables for which such information is not otherwise available. Sometimes univariate analysis is sufficient to meet major study objectives. For instance, in a diagnostic survey of clients intended to indicate strengths and weaknesses of a particular project, univariate analysis of responses to questions about perceptions of and reactions to various project opera- tions would constitute an important part of the analysis. The simplest form of single-variable analysis is to count the number of cases in each category. However, the resulting count, called a frequency distribution, usually is not very interesting without additional s

View the publication