IMMPACT Toolkit- Sampling at Service Sites (SSS)

Publication date: 2007

Toolkit A guide and tools for maternal mortality programme assessment MODULE 4, Tool 1 Sampling at Service Sites (SSS) Version 2.0 SSS SSS SSS SSS SSS SSS SSS SSS SSS Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools Immpact © 2007 University of Aberdeen List of Acronyms DHS Demographic and Health Survey Epi Info epidemiology data entry and analysis software GFR general fertility rate LTR lifetime risk MMR maternal mortality ratio MRFRA mortality rate amongst females of reproductive age MMRate maternal mortality rate PMR proportionate mortality ratio SSS Sampling at Service Sites WHO World Health Organization Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools Immpact © 2007 University of Aberdeen Module 4, Tool 1 – SSS: Table of Contents Introduction .4 4(1).1 What is SSS? .4 4(1).2 Why use SSS? .5 4(1).3 Limitations of SSS.5 4(1).4 Can SSS be used in different contexts?.6 4(1).5 Using SSS .7 Step 1: Identify comparator data set.7 Step 2: Identify suitable service sites and collect usage data .7 Step 3: Obtain permissions .8 Step 4: Prepare for the study .8 Step 5: Collect data .9 Step 6: Data management and analysis .10 Step 7: Disseminate findings and recommendations .11 4(1).6 Follow-up to study.12 4(1).7 Budget implications of using SSS .12 4(1).8 SSS data collection instruments.13 SSS Sample Questionnaire .13 SSS Sample Analysis Plan .17 References.21 Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 4 Immpact © 2007 University of Aberdeen INTRODUCTION This tool describes a technique for measuring rates of maternal mortality in the community which offers potential cost and time savings over traditional house-to-house type surveys. The information given here is intended to help anyone in the process of commissioning a study to decide whether Sampling at Service Sites (SSS) will be a useful method; and to help anyone planning a measurement of maternal mortality using the SSS method to design and execute a survey. The need to provide accurate statistics on health in developing countries has been highlighted by the WHO Health Metrics Network (www.who.int/healthmetrics/en/) However, no simple, low-cost method exists for estimating the levels of maternal mortality in countries where there is no reliable system of vital registration. The problems and most of the commonly used methods are outlined in the WHO publication Maternal Mortality in 2000 (WHO, 2004). Most practical methods involve gathering information through some form of survey, but to achieve a satisfactory level of precision in the estimates this normally requires sample sizes of many thousands (depending on the prevailing rates of maternal deaths and levels of fertility). Example sample sizes required in various scenarios are shown in section 4(1).5, Step 2. Surveys to measure rates of maternal mortality have often been added to other major surveys such as Demographic and Health Surveys (DHS) (/www.measuredhs.com/aboutsurveys/dhs) (Stanton et al, 1997) or censuses (Stanton et al, 2001), which take place at typically five- or ten-year intervals. Even in relatively large surveys such as the DHS, the sample sizes used usually do not permit useful estimates to be made at sub-national levels. Where it is desired to measure the rate of maternal mortality at other times or at sub-national levels, large surveys must be carried out specifically for this purpose. Although only a small amount of information needs to be gathered from each respondent to measure maternal mortality, carrying out a house-to-house type of survey specifically for this purpose carries with it many of the same cost and logistical overheads as administering a much wider-ranging survey. SSS has been developed as an alternative survey method to improve efficiency and hence save costs compared to traditional survey methods. 4(1).1 What is SSS? SSS is a survey method for capturing data on maternal deaths which uses opportunistic sampling at health facilities, markets or other sites. The questionnaire uses the Direct Sisterhood Technique to ask about maternal deaths. The Sisterhood Technique (Graham et al, 1989; WHO, 1997) is similar to modules used to measure maternal mortality in DHS surveys, and involves asking individuals about maternal deaths amongst their sisters who reached age 15 or above, born to the same mother. It is based on the premise that people are likely to know if a sister has died whilst pregnant, during childbirth or soon after being pregnant. It also has the advantage that one respondent can give information about several sisters, the efficiency being greatest in countries with very high fertility. The Direct Sisterhood Technique attempts to gain current estimates of mortality by asking about deaths during the five years before the survey. It is therefore an adaptation of the original Indirect Sisterhood Technique which asks about any past deaths. Adult women are usually selected as informants but the technique can also be applied with men as informants. Respondents are asked questions about: • the number of adult sisters they have ever had; • whether any of these sisters have died whilst pregnant, during childbirth or within 42 days of the end of a pregnancy; and • their own socio-economic status. Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 5 Immpact © 2007 University of Aberdeen Their responses are then compared to the wider community through data collected from another reliable source such as DHS or a census, and are used to estimate the level of maternal mortality prevailing in the community over the five years before the survey, using a number of indicators: maternal mortality ratio (MMR), maternal mortality rate (MMRate), lifetime risk (LTR) and the proportionate mortality ratio (PMR). These are explained further in Step 5 (section 4(1).5). Although the method has been developed primarily to measure maternal mortality using health facilities and markets as the ‘service sites’, in principle it could also be used at many other types of site where people gather. It can also be used to measure other socio-economic and health-related variables of interest. In the initial Immpact surveys using SSS, major health facilities (large health centres and district hospitals) were selected as sites with many potential survey respondents. Healthy women aged 15–49 attending antenatal, postnatal, child health or family planning clinics, or accompanying a sick person at the outpatients department, were interviewed as they waited to be seen by the medical staff. 4(1).2 Why use SSS? The advantages of using SSS are: 1) Rapid respondent identification. The method reduces the financial and logistical overheads of carrying out a major community survey by letting the respondents come to the interviewers rather than sending the interviewers out to the respondents' houses as in a traditional survey. Achievable sample sizes depend on how many service users there are. In countries with a relatively low use of health services (e.g. in Burkina Faso), marketplaces have proved to be a promising alternative type of service site. 2) Rapid and efficient interviews. The questions for each respondent are relatively few (our questionnaires took typically two to ten minutes to administer). SSS therefore minimizes the time between interviews and is an efficient way of capturing this kind of information. In our surveys in Ghana, teams of four interviewers plus one supervisor were able to interview up to 170 women in a day. 3) Reduced cost and time. Cost and/or time per respondent in a survey will be reduced if interviewers are not sent out to people’s homes. This means either that a survey can be carried out more quickly and with lower financial resources, or that a greater sample size can be obtained for a given sum of money. Potential cost savings result from interviewers having a higher average rate of interviews per day than they would in a house-to-house survey and from reduced travel and supervision costs because interviews take place in a relatively small number of centralized locations (e.g. district hospitals). 4) Greater precision of maternal mortality estimate Greater sample size usually leads to better precision in the resulting estimate of the rate of maternal mortality. 4(1).3 Limitations of SSS SSS, by its nature, selects only from those people who use a particular service (e.g. health care). Therefore it does not produce a random sample from the community. However, it should be remembered that the survey is measuring rates of maternal mortality amongst the sisters of respondents, who may or may not themselves be service users. Nevertheless, we can expect some degree of similarity between siblings, so it is to be expected that there will be some bias in the respondents’ sisters compared to the community as a whole. Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 6 Immpact © 2007 University of Aberdeen Questions about the respondents' own socio-economic status included in the survey can tell us something about the degree of bias. A decision can then be made as to whether to accept this or to correct for bias by weighting respondents with different socio-economic characteristics accordingly. So far this does not seem to have been necessary in the surveys that Immpact has conducted, and attempts to apply weighting have resulted in little change to the overall estimates of maternal mortality rates obtained. In cases where there is a small but not negligible bias in respondents, it will, in practice, often be possible to predict the direction of bias in respondents and sisters. For example, people attending hospitals in Ghana tend to be richer or more urban than the community as a whole. Assuming that maternal mortality is lowest in the richer, urban population, it would therefore be possible to regard an estimate from an SSS survey as representing a slight underestimate of the true rate of maternal mortality in the wider community. Conversely, in cases where poorer, more rural people use the selected service sites, SSS estimates could be thought of as probably representing a slight overestimate of the true rate of maternal mortality in the community as a whole. 4(1).4 Can SSS be used in different contexts? In principle, SSS could be used anywhere and on any scale. However, in practice it makes sense to use it only in situations where data capture can be carried out more efficiently than it would be in a house-to-house type of survey. For measuring maternal mortality, sample sizes in the thousands are usually required, so the feasibility of using SSS also depends on having suitably large service sites available to carry out the survey. In most cases, the method will need adapting to a particular setting, and the research team should take the following considerations into account. The desirable characteristics of service sites are: 1. Large numbers of potential respondents (e.g. healthy women aged 15–49) are present. 2. Potential respondents at each site represent a wide cross-section of society, or a set of service sites between them would give a wide cross-section of society. 3. The sites are suitable places to ask respondents questions about maternal deaths amongst their sisters. 4. Respondents would be willing to spend 5–15 minutes answering questions in that setting. 5. Management at the site is cooperative and sympathetic to the aims of the survey. Tip: So far Immpact has found SSS to work well at large health facilities in Ghana, where antenatal care coverage is 90%. However, in Burkina Faso, where antenatal care coverage is only 72%, market places have been found to provide greater numbers of respondents. Since access to services may also depend on the rainy season in many countries, this should also be considered in selecting the best time of year to carry out a survey. In a house-to-house survey, the only limit to the sample size is the size of the population. It must be remembered that in SSS there is a practical upper limit set by the number of service users there are and the duration of the survey. The efficiency of the SSS method is gained by restricting sampling to a fairly small number of high volume service sites (e.g. district hospitals). In practice, there will often be only a few of these available and it may not be possible to achieve the required sample size by sampling at all of these. A calculator is presented in Step 2 of section 4(1).5 to help decide whether enough respondents could be obtained from the available service sites in a reasonable time. Since there are practical limits on the sample size that can be achieved with SSS it can be expected that SSS will work best in situations where the required sample size is not too large, i.e. where there are high rates of Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 7 Immpact © 2007 University of Aberdeen maternal deaths or the degree of precision required is not too great, and in areas where there are high rates of usage of the chosen type of service site. 4(1).5 Using SSS Step 1: Identify comparator data set The first step involves the identification of existing data sets to determine the representativeness of the SSS sample. The comparator data set provides a recent description of the population with regard to basic demographic and socio-economic characteristics. In many countries, such sources of population-based data are available through, for example, the demographic census or household surveys such as the DHS. Although the sample sizes of many such surveys are clearly inadequate for measuring current levels of maternal mortality, they are able to provide robust estimates of the population profile. At least one good quality comparator data set should be identified to assess the representativeness of the respondents captured in an SSS survey. Where the SSS respondents have a similar profile to the wider community, this will be helpful in interpreting the survey results. If a large bias is observed it might also be possible to weight respondents from the under-represented sections of the community using some form of wealth index such as those used by the DHS or the World Bank (Rutstein and Johnson, 2004). If no comparator data set is available, consider carrying out a small community survey. The sample size needed to establish reliable estimates of the proportions of people with different ages, education and possessions should be very much less than that required for a survey of maternal deaths. A popular method for rapid collection of community data in developing countries is the Epi Cluster method. Information about this and other methods for collecting this kind of data is available from the University of California, Department of Epidemiology website (www.ph.ucla.edu/epi/rapidsurvey.html). Once a suitable comparator data set has been identified, a set of questions from that survey should be selected to reflect the socio-economic status of the respondents. These should be reproduced exactly in the SSS survey questionnaire so that an accurate comparison can be made. Examples of the kind of questions to include are given in the sample questionnaire in section 4(1).8. We recommend that 5–10 such questions be included to obtain a clear picture of the socio-economic profile of the survey respondents. Step 2: Identify suitable service sites and collect usage data The second step involves gathering information on the size, types of clinic and location of major health service delivery points in the study region. To enable the greatest efficiency savings to be gained from SSS, these services need to be reasonably high volume. In most of our surveys, district hospitals were chosen as suitable sites, providing a full range of outpatient clinics and with wide population coverage, though other types of service site may be more suitable in other settings. The target respondent group was women aged 15–49 years, and thus all hospitals in each study region providing the following outpatient clinics were selected: antenatal care, child welfare (immunization and growth monitoring), family planning, and sick child clinics. On the basis of routine health statistics, it was possible to estimate the number of outpatient contacts expected. At this stage in the planning process, an assessment should be made as to whether it will be feasible to carry out an SSS survey in a particular situation in terms of the number of respondents required and the number available at the chosen service sites. Calculator tools can be used online at the Immpact website that allow you to firstly calculate the approximate number of respondents that will be required and secondly to plan how many facilities will be required and how many weeks sampling would need to be carried out to obtain the required sample size. The calculations used in both tools involve a number of approximations and assumptions so can only give approximate answers. Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 8 Immpact © 2007 University of Aberdeen Calculator for the number of survey respondents required can be used online at: (www.abdn.ac.uk/~ogy403/toolkit/module4/sss/calc/) This calculator allows you to estimate the approximate number of survey respondents that would be required to carry out an SSS survey to measure maternal mortality using the direct sisterhood technique. On the online version of the form, enter the information requested then click Calculate to obtain the estimated sample size. The entries can be changed and the Calculate button clicked again any number of times. Calculator for planning how the required number of survey respondents can be obtained from the chosen type of service sites (www.abdn.ac.uk/~ogy403/toolkit/module4/sss/calc/) This calculator allows you to plan a sampling strategy. Greatest efficiency will be achieved by including the largest service sites first (e.g. district hospitals) then making up the required number of respondents by adding smaller sites (e.g. rural health centres) until the required sample size is obtained. For each site or type of site, enter the information requested in the first three columns. The number of respondents it is expected to obtain from each facility or type of facility and total number of respondents expected is then calculated when the Calculate button is clicked on the online form. More facilities can be added or any of the figures already entered can be changed and the Calculate button clicked again until the required number (see sample size calculator above) has been obtained. No allowance is made in the calculator for repeat visits. If the survey is carried out at any location for more than a few weeks it is to be expected that some of the people attending will already have been interviewed. They should not be interviewed again and allowance should be made for this when estimating numbers of respondents expected. Step 3: Obtain permissions Permissions should be sought from the collaborating institutions’ ethics committees, from the regional health administrations in the country, and from the participating hospitals and health centres. Step 4: Prepare for the study The fourth step in the application of the SSS approach to estimating maternal mortality involves all the essential work up to data capture. Design and pre-test questionnaire A questionnaire should be designed and pre-tested, including three main sections: the clinic attendance, the characteristics of the respondent, and the Direct Sisterhood Technique questions to elucidate survival status of all the respondents’ adult sisters, with additional questions on dead sisters, including date, place and whether death occurred during pregnancy, delivery or during the six weeks after the end of pregnancy. The latter is the standard question used in the Sisterhood Method to identify maternal deaths and thus a time-of-death definition is utilized. A sample questionnaire, similar to those used in the Immpact surveys, is included in section 4(1).8. In part 4 of the questionnaire, the interviewer should fill in one column for each of the respondent's adult sisters. In some cases this will be none and in most cases will be no more than three. The form is set up to accommodate up to six sisters so in most cases it will not be necessary to complete the whole form. Extra sheets should be available for the few cases where respondents have more than six sisters. Whilst it is not necessary to include all of the socio-economic questions, we recommend that at least five such questions be included. These should exactly match the wording in the comparator data source chosen (see Step 1). Additional questions can be added to the questionnaire to address particular research needs. We have Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 9 Immpact © 2007 University of Aberdeen found it more efficient to add some types of further question as modules which may be asked, for example, just to women who report a maternal death or to one in every five respondents. Train fieldworkers Fieldworkers need to be recruited and trained in the application of the questionnaire and in appropriate interpersonal skills for approaching and interviewing respondents in health service sites. Once agreement on the appropriate way to ask questions has been reached, guidelines should be prepared for the interviewers to refer back to. Tip: We found a training workshop of four to five days was appropriate. This included one morning in a health facility both as training for the interviewers and to pre-test the questionnaire. The training workshops also included a discussion on how to translate the questions into the local language(s). More information on adaptation and translation is included in Module 5, Technical Annex C. Conduct orientation visits Orientation visits should be made to each participating hospital to assess the clinic layout and patient flow in order for the interviewing to cause minimal disruption to the service provision. Advice should be sought from clinic personnel on the recruitment process and on suitable venues in which to conduct private interviews. Clear guidelines should be provided so that the interviewers know not to approach women who are ill or who need to attend to sick children or adults they are accompanying. Step 5: Collect data The fifth step is the conduct of the data capture at the chosen service sites. Teams of interviewers should visit each health facility on those days when the target clinic is held, and recruit as many respondents as possible. Team supervisors should be responsible for allocating interviewers between clinics to cope with busy periods of the day, so as to minimize the number of women leaving without being approached. The calculator in Step 2 assumes that all potential respondents will be approached for interviews, which in turn assumes that enough interviewers are assigned to cope with the busiest periods. We found interviewers could interview up to approximately 40 respondents each per day. Since the daily interview rate depends on how busy the site is, the best efficiency will be achieved if the number of interviewers at each site can vary from day to day. In Immpact’s experience, four interviewers were needed on the busiest days (which could often be predicted) to reach all potential respondents, but on most other days fewer were needed. Greater efficiency may be achieved by having fewer interviewers and accepting that some potential respondents will be missed during the busiest times; however, allowance must be made for this in calculating the number of weeks’ sampling that will be required. Figure 4(1).1 shows the numbers of respondents per day obtained at a number of health facilities (district hospitals and large health centres) during one of our surveys, ordered from lowest to highest. As can be seen from the graph, very high numbers (>100) were obtained only on a few days. We estimate that by having used four interviewers per facility, most of the potential respondents were interviewed and that those who were missed or chose not to answer the survey were fewer than 10%. The supervisor should also cover quality control of the data capture, observing each interviewer at regular intervals and cross-checking the questionnaires before they are submitted for data entry, passing any queries back to the interviewer concerned. Immpact Toolkit: a guide and tools for maternal mortality programme assessment Figure 4(1).1: Daily totals of respondents per health facility Step 6: Data management and analysis Data entry templates must be set up using suitable software. One suitable package is Epi Info which can be downloaded free from the CDC website (www.cdc.gov/epiinfo/). The older version (Epi 6) is available via a link from the same web page and can be run on even a very basic PC. A regime for regular data transfer from the field sites to the office should be planned (e.g. weekly). Entering and checking at least some of the data during the first week of the survey can help to identify mistakes in field procedures. Data should be entered, checked for consistency and cleaned (i.e. internal inconsistencies identified and checked) as the survey progresses. It is possible to compare the profile of respondents obtained up to any point with the comparator data source (see Step 1). Hence, if the respondents do not seem to be representative enough, a decision can be taken to move further sampling to different locations (e.g. to more rural health centres). Often this will entail some loss in efficiency because rural health centres have fewer patients, so a balance must be found. Figure 4(1).2 demonstrates an example comparison between a fictional SSS sample and comparator data source. As can be seen in the graph, the survey respondents in this example are considerably richer than the population in general for the area of interest. To obtain a fairly representative sample it would be useful to carry out the survey in more rural locations in the following weeks. Tip: We have found double entry (two people independently entering the data, then a comparison made of the two data files) as the most efficient way of eliminating data entry errors. Epi Info includes routines to facilitate this process. SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 10 Immpact © 2007 University of Aberdeen Immpact Toolkit: a guide and tools for maternal mortality programme assessment 0 10 20 30 40 50 60 70 80 90 Ow ns a TV Ow ns a car Secondary education or higher Has piped w ater inside the house House has brick, stone or metal w alls % DHS Sample (weeks1+2) Figure 4(1).2: Example of comparison between sample and comparator data set (DHS) A data analysis plan should be drawn up before the survey commences and cross-checked against the questionnaire to ensure all necessary questions have been asked. Analysis will normally include: 1. Comparison of the proportions of respondents with the proportions in the comparator data source for each of the demographic and socio-economic questions in the questionnaire. 2. Maternal mortality ratio (MMR) – the number of maternal deaths per 100,000 live births. 3. Maternal mortality rate (MMRate) – the rate of maternal deaths per woman aged 15–49 per year. 4. Lifetime risk (LTR) – the probability of a woman dying a maternal death. This depends on both the MMR and the fertility rate. 5. Proportionate mortality ratio (PMR) – the proportion of all deaths of women aged 15–49 which are maternal. Many other analyses of the data set are possible, particularly when additional questions have been added to the questionnaire to address particular needs. A more detailed example of a Sample Analysis Plan is shown in section 4(1).8. Step 7: Disseminate findings and recommendations The final step in the application of SSS is to interpret the findings and utilize appropriate local dissemination mechanisms. This may include local and regional workshops, policy briefs and other forms of publication. An important step in the interpretation is to compare the results with those from other studies in or near to the same area and to consider the reasons for any differences (e.g. changes over time, artefacts of methods, differences in the area or population covered). Account should be taken at the planning stages of who the end users of the findings will be. Interested parties should be considered and possibly involved in the planning process to ensure that an estimate of maternal mortality produced using the SSS method will meet the overall objectives of the survey. If SSS is being applied as part of a broader evaluation on maternal health care, the findings from this arm of the evaluation should be integrated with other study results. SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 11 Immpact © 2007 University of Aberdeen Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 12 Immpact © 2007 University of Aberdeen 4(1).6 Follow-up to study Since SSS usually measures mortality over a retrospective period prior to the survey (usually five years) it is usually necessary to wait five years before repeating the survey. If a follow-up survey is conducted sooner and the retrospective periods overlap, the survey is unlikely to show any change and would therefore be a wasted effort. In most situations, it would be preferable to use the same sites for a follow-up survey since any bias due to the choice of sites would be similar in both surveys, and therefore any differences in results are likely to be real changes in the maternal mortality rates. It is not essential, however, that the same sites are used on both occasions, provided the sets of sites as a whole in each survey are a reasonably representative sample of the population. If very different sites are used in two sequential surveys (e.g. health centres in one survey and markets in another), this should be considered when interpreting the results. 4(1).7 Budget implications of using SSS Table 4(1).1 shows recurrent costs associated with use of SSS. No capital costs (such as vehicles, buildings or equipment for general research work) are included in this table. The table shows the resources required for a survey of around 12,000 women. Table 4(1).1: SSS resource requirements Quantity Time Supplies • 12,200 copies of the questionnaire • Sufficient copies of the survey manual for one per interviewer/supervisor • CD-ROMs for storing data Not applicable • Team of interviewers (typically 8–20) • 3 days training (including 1 day pre-testing the questionnaire in a typical service site) • Typically 6–14 weeks interview time • Supervisors – one per service site • Typically 6–14 weeks • Senior supervisors (one or two) • Visits to local headquarters or study sites to announce intention to carry out the survey and obtain permissions (2–4 weeks) • Typically 6–14 weeks for the survey • Data entry staff (two or three) • Typically 6–14 weeks (assuming sufficient data entry staff to keep pace with data collection) • Data entry usually runs 1–2 weeks behind collection Personnel • Drivers • Depends on organization of the survey Travel and communication • Vehicle to make preliminary visits to set up the survey • Vehicle to travel to interview sites and to health facilities • Travel and subsistence costs for survey staff, supervisors and drivers • Phone calls • 2–4 weeks • Typically 6–14 weeks Building operation and maintenance • Data entry facility required and somewhere to store the questionnaires securely • Not applicable Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 13 Immpact © 2007 University of Aberdeen 4(1).8 SSS data collection instruments This section contains the following instruments: 1. SSS sample questionnaire; 2. SSS sample analysis plan. SSS Sample Questionnaire Service Site Sampling Questionnaire 1. BACKGROUND 1.1 District Coding manual 1.2 Health facility FFFF, Coding manual 1.3 Staff code SS, Coding manual 1.4 Date of interview DDMMYYYY 1.5 Time of start of interview HHMM, 24 hour clock 1.6 Type of clinic attended by respondent 1=antenatal; 2=postnatal; 3=child welfare; 4=family planning; 5=accompanying sick child or adult; 6=other 2. RESPONDENT DETAILS: Firstly I would like to ask you a few questions about yourself and your household. 2.1 What is your name?.……………………………………….………………………….…………………………… 2.2 How old are you? (88=don’t know) 2.3 What is the name of the town or village where you usually live? ….…………………………………………………………………………………….…… 2.4 In which district do you usually live? (coding manual; 000 = not in coding manual; 888 = don’t know) 2.5 In which region do you usually live? (coding manual; 00=another country; 88=don’t know) 2.6 What is the highest level of schooling you attended? 1=none; 2=pre-school/primary; 3=middle/JSS; 4=secondary or higher including vocational; 8=don’t know 2.7 What is the main construction material used for the outer walls of your dwelling? 1=mud/mud bricks/earth; 2=cement blocks/concrete; 3=other; 8=don’t know 2.8 What is the main material used for the roof of your dwelling? 1=thatch/palm leaves/raffia/bamboo/mud/mud bricks/earth/wood; 2=corrugated metal sheet/slate/asbestos/cement/concrete/roofing tiles; 3=other; 8=don’t know Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 14 Immpact © 2007 University of Aberdeen 2.9 What is the main source of drinking water for your household? 1=pipe-borne inside; 2=pipe-borne outside/tanker supply; 3=well/bore-hole/spring/rain water/river/stream/dugout/pond/lake/dam; 4=other; 8=don’t know 2.10 What type of toilet is used by your household? 1=private facility; 2=public toilet; 3=no facilities (bush/beach/field); 4=other; 8=don’t know 2.11 What is the main source of lighting for your dwelling? 1=electricity; 2= kerosene lamp; 3=other; 8=don’t know 2.12 Does your household own a radio? 1=yes; 2=no; 8=don’t know 2.13 Does your household own a bicycle? 1=yes; 2=no; 8=don’t know 2.14 Does your household own a motorcycle or car? 1=yes; 2=no; 8=don’t know 3. RESPONDENT’S ADULT SISTERS: Now I would like to ask you some questions about your sisters. 3.1 How many sisters have you ever had (sisters who were born to the same mother as you) who became adults (reached their 15th birthday) (not including yourself)? 00=none; 88=don’t know 3.2 How many of these sisters are alive now? 00=none or not applicable; 88=don’t know 3.3 How many of these sisters are dead now? 00=none or not applicable; 88=don’t know Check that 3.1 = 3.2 + 3.3. If no adult sisters draw a diagonal line through the rest of section 3 and go to section 4. Write down the names of all of the respondent’s adult sisters in order of age. Check that total agrees with 3.1. Sister 0 1 S ister 0 2 S ister 0 3 3.4 What is the name of your eldest (next) sister? ………………. ………………. ………………. 3.5 How many children has [name] had in the last 5 years who were alive at the moment of birth? Enter the number; 0=none; 8=don’t know 3.6 Is [name] alive? 1=yes; 2=no; 8=don’t know 3.7 How old is [name]? 777=no longer alive; 888=don’t know 3.8 What is the name of the village where [name] lives or lived at the time she died? ………………. ………………. ………………. 3.9 Which district does [name] live in or did she live in at the time she died? Coding manual; 000=not in coding manual; 888=don’t know Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 15 Immpact © 2007 University of Aberdeen 3.10 Which region does [name] live in or did she live in at the time she died? Coding manual; 00=another country; 88=don’t know If alive or don’t know, draw a diagonal line through the rest of the column and go back to 3.4. If not alive, go to 3.11. 3.11 How old was [name] when she died? 888=don’t know 3.12 Did [name] die in the last 5 years? 1=yes; 2=no; 8=don’t know If aged less than 50 at death go to 3.13. Otherwise draw a diagonal line through the rest of the column and go to 3.4 for the next sister. 3.13 Was [name] pregnant when she died? 1=yes; 2=no; 8=don’t know 3.14 Did [name] die during childbirth? 1=yes; 2=no; 8=don’t know 3.15 Did [name] die within 6 weeks of the end of pregnancy? 1=yes; 2=no; 8=don’t know Repeat from question 3.4 for next sister in next column. If no more sisters draw a diagonal line through remaining SISTER columns and go to section 4. Write down the names of all of the respondent’s adult sisters in order of age. Check that total agrees with 3.1. Sister 0 4 S ister 0 5 S ister 0 6 3.4 What is the name of your eldest (next) sister? ………………. ………………. ………………. 3.5 How many children has [name] had in the last 5 years who were alive at the moment of birth? Enter the number; 0=none; 8=don’t know 3.6 Is [name] alive? 1=yes; 2=no; 8=don’t know 3.7 How old is [name]? 777=no longer alive; 888=don’t know 3.8 What is the name of the village where [name] lives or lived at the time she died? ………………. ………………. ………………. 3.9 Which district does [name] live in or did she live in at the time she died? Coding manual; 000=not in coding manual; 888=don’t know 3.10 Which region does [name] live in or did she live in at the time she died? Coding manual; 00=another country; 88=don’t know Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 16 Immpact © 2007 University of Aberdeen If alive or don’t know, draw a diagonal line through the rest of the column and go back to 3.4. If not alive, go to 3.11. 3.11 How old was [name] when she died? 888=don’t know 3.12 Did [name] die in the last 5 years? 1=yes; 2=no; 8=don’t know If aged less than 50 at death go to 3.13. Otherwise draw a diagonal line through the rest of the column and go to 3.4 for the next sister. 3.13 Was [name] pregnant when she died? 1=yes; 2=no; 8=don’t know 3.14 Did [name] die during childbirth? 1=yes; 2=no; 8=don’t know 3.15 Did [name] die within 6 weeks of the end of pregnancy? 1=yes; 2=no; 8=don’t know [If more than 6 sisters continue on an extra sheet] 4. END OF INTERVIEW THANK RESPONDENT FOR HER TIME AND END THE INTERVIEW Interviewer name:……………………………………………………………………………………………………………. Checked by supervisor:……………………………………………………………………………………………………… Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 17 Immpact © 2007 University of Aberdeen SSS Sample Analysis Plan This section contains a fictional SSS analysis plan for the country of ‘Myland’ (fictional country). Data analysis plan for survey to measure the maternal mortality ratio for ‘Myland; using SSS Introduction The survey has been designed to measure the maternal mortality ratio (MMR) and related indicators in Myland, averaged over the period 2001–2005, i.e. five years prior to the survey. Background information A sample size of 10,700 respondents has been used which is intended to achieve a 95% confidence interval (i.e. precision) of +/- 20% in the estimate of MMR based on the following prior information: General Fertility Rate (GFR) = 120 per 1000 women aged 15–49 per year Expected MMR = 700 maternal deaths per 100,000 live births Mean number of sisters per respondent = 2.3 (from small scale pilot survey) An exposure adjustment factor of 0.9 was used in calculating the required sample size to allow for the fact that not all sisters would have been within the age range 15–49 throughout the five year reference period prior to the survey. The survey will be carried out at all 15 government district hospitals in Myland for a period of three weeks at each location. Data entry The data will be double-entered using a template designed using Epi Info. All discrepancies will be identified and corrected by the junior data manager. Data cleaning Data will be checked for internal consistency using a cleaning program developed in Epi Info. In particular, checks will be made that: 1. The date is correct, i.e. the form numbers fall in the expected date order; 2. The staff code and hospital code are correct, i.e. that they are consistent with other forms in the batch and that that interviewer was working in that hospital on that day; 3. The clinic code is one of the permissible codes; 4. The age of the respondent is in the range 15–49; 5. The respondent’s district is consistent with the region; 6. The number of sisters is equal to the number of live sisters plus the number of dead sisters; 7. One column has been filled in for each adult sister; Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 18 Immpact © 2007 University of Aberdeen 8. The sisters’ ages are all 15 or over; 9. The sisters’ home district and region should be consistent with each other; 10. If the sister is dead, the age is recorded as not applicable, an age is recorded for age died and the question whether she died in the last five years is answered; 11. If the sister is alive, the ‘sister’s age died’ is recorded as not applicable; an age is recorded for current age and the question whether she died in the last five years is completed as not applicable; 12. If the sister is dead and the age at death was less than 50, the three maternal death questions are answered, otherwise these are entered as not applicable; 13. No more than one of the three maternal death questions is answered ‘yes’; 14. The numbers of live and dead sisters recorded in the individual sister columns are consistent with the totals given in the previous section of the form; 15. For all questions the response is a valid number, date, code, or the code for ‘don’t know’ or ‘missing’. Additionally, range checks will be made on some of the variables to look for likely errors. The following range checks will be made: 1. The total number of sisters is 9 or fewer; 2. The total number of dead sisters is 5 or fewer; 3. The number of live births in the last 5 years for any sister is 6 or fewer; 4. The sister’s age or age at death is less than 100; 5. The sister’s age is no more than 35 years different from the respondent’s own age. Whilst it is theoretically possible for these ranges to be exceeded, these will be used to prompt re-examination of the questionnaire forms. If it is not clear that the data are in error they will be retained even though they may exceed these ranges. Where it is possible to identify an error, corrections will be made to the data set and a record of such changes kept. An example of this is where the total number of sisters is recorded as five but the total number of live sisters is shown as two and the total number of dead sisters is shown as two. Examination of the individual sister records shows three live sisters and two dead sisters. It is reasonable to assume that the total number of live sisters has been recorded wrongly and this will be corrected to three. Where there is clearly an error (e.g. the sister’s age is recorded as 200) but no evidence for what the correct value should be, the data will be replaced by the code for a missing value. A record of any such changes will be kept. At the end of the data cleaning process a committee made up of the senior supervisors, data manager and another senior member of the project staff will consider any unresolved cases and a decision will be made as to whether the entries should be changed or replaced with missing values in each case. A record will be made of any such cases. Data quality checks The quality of the data will be checked by an independent assessor (e.g. an advisor to the project) and a short report written. This will include such findings as the tendency for ages to be reported to the nearest multiple of 10 years, the proportions of missing values, systematic differences in the results from different interviewers, comparison of age distributions and fertility rates with those given in other sources such as the DHS or census, and any inconsistencies in the data remaining after data cleaning has been completed. Consideration will be given to the implications of the assessor’s report for the intended outputs of the survey and the use that will be made of these. In the event that the data appear to be very unreliable, consideration will be given as to the usefulness of completing the analysis. Options in the case of less severe problems include returning to data cleaning, entering the data again or collecting some additional data to verify unexpected results. Immpact Toolkit: a guide and tools for maternal mortality programme assessment Data analysis The characteristics of the respondents will be described in terms of the proportions with each of the characteristics recorded in Part 2 of the questionnaire (respondent’s demographic and socio-economic details). These will be compared with the equivalent proportions for women aged 15–49 from the comparator data source (the national demographic census for 2000). These comparisons will be used in the survey report to aid interpretation of the survey findings. The characteristics of the sisters: mortality rate amongst females of reproductive age, fertility rate estimated from the number of births in the last five years, the maternal mortality ratio (MMR), maternal mortality rate (MMRate), lifetime risk (LTR) and proportionate mortality ratio (PMR) will be calculated and compared with any available published figures, e.g. WHO-modelled estimates, studies from demographic surveillance sites within or near to the survey area, DHS estimates, and survey results published in journal papers. Details of the calculations are given below. Mortality rate amongst females of reproductive age (MRFRA) The total exposure (number of sister-years within the age range 15–49) will be calculated assuming: (i) that sisters had their last birthday exactly six months before the survey, e.g. a sister whose reported age was 23, was actually aged 23.5 years at the time of the survey; (ii) that sisters who died in the last five years died exactly 2.5 years before the survey (where five years was the chosen reference period). Using these approximations, the exposure for each sister can be obtained from the following lookup table. Exposure (years) aged 15–49 Stated age or age died Live sisters Dead sisters (death in last 5 years) Dead sisters (death more than 5 years ago) 15 0.5 0.5 0 16 1.5 1.5 0 17 2.5 2.5 0 18 3.5 2.5 0 19 4.5 2.5 0 20–49 5 2.5 0 50 4.5 2 0 51 3.5 1 0 52 2.5 0 0 53 1.5 0 0 54 0.5 0 0 55+ 0 0 0 The mortality rate (per 1000 women years at risk) will then be calculated using the following formula: 1000 exp 5 ×= sistersamongstosure yearslastinsistersamongstdeathsMRFRA SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 19 Immpact © 2007 University of Aberdeen Immpact Toolkit: a guide and tools for maternal mortality programme assessment General fertility rate (GFR) Exposure will be calculated as for mortality rate amongst females of reproductive age, (see above). The mean number of births per 1000 women of reproductive age (15–49) per year over the last five years will then be calculated using the following formula: 1000 exp 5 ×= sistersamongstosure yearslastinsistersamongstbirthstotalGFR Maternal mortality ratio (MMR) The number of maternal deaths per 100,000 live births over the last five years will be calculated using the following formula: 100000 5 5 ×= yearslastinsistersamongstbirthslive yearslastinsistersamongstdeathsmaternalMMR Maternal deaths are any sister who died whilst pregnant, giving birth or within 42 days of the end of a pregnancy (time-related definition). Maternal mortality rate (MMrate) Total exposure will be calculated as for the mortality rate amongst females of reproductive age (see above). The MMrate (i.e. the number of maternal deaths per 1000 women years of exposure aged 15–49) will then be calculated using the following formula: 1000 exp 5 ×= sistersamongstosure yearslastinsistersamongstdeathsmaternalMMrate Lifetime risk (LTR) The probability of a woman dying a maternal death, assuming she has babies at the prevailing age specific fertility rates for the area throughout her life, will be calculated using the formula: ( )TFRMMRLTR −−= 11 For this calculation MMR is the maternal mortality ratio expressed as a fraction (i.e. per live birth, not per 100,000 live births). TFR is the total fertility rate – the number of live births a woman would be expected to have during her lifetime if she lives beyond child-bearing age. This will be obtained from the census report. Proportionate mortality ratio (PMR) The proportion of deaths to reproductive aged women (15–49) which are maternal deaths will be calculated using the following formula: yearslastinsistersamongstdeaths yearslastinsistersamongstdeathsmaternalPMR 5 5= Deaths amongst the sisters in the denominator include maternal deaths. SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 20 Immpact © 2007 University of Aberdeen Immpact Toolkit: a guide and tools for maternal mortality programme assessment SSS SSS SSS SSS SSS SSS SSS SSS SSS Module 4: Evaluation tools 4 : SSS : 21 Immpact © 2007 University of Aberdeen Data archiving The survey questionnaires will be stored in the project office for the duration of the project. Subsequently they will be transferred to the Institute data archive room and kept for a period in line with institute policy. Two copies of the computer files will be kept on CD-ROM in the project office and two copies will be kept by other members of the survey team. In the event of any corrections being made to the data the data manager will update the copies held in the office and supply updated copies to the other two data holders. REFERENCES Graham W, Brass W, Snow RW (1989) Estimating maternal mortality: the sisterhood method. Studies in Family Planning, 20 (3), 125–35. Rutstein SO, Johnson K (2004) The DHS Wealth Index. Calverton MD: ORC Macro (DHS Comparative Reports no. 6). Stanton C, Abderrahim N, Hill K (1997) DHS maternal mortality indicators: an assessment of data quality and implications for data use. Calverton, MD, Macro International Inc., (Demographic and Health Surveys Analytical Report No. 4). Stanton C, Hobcraft J, Hill K, Kodjogbe N, Mapeta WT, Munene F, Naghavi M, Rabeza V, Sisouphanthong B, Campbell O (2001) Every death counts: measurement of maternal mortality via a census. Bulletin World Health Organ, 79 (7), 657–64. WHO (1997) The sisterhood method for estimating maternal mortality: guidance notes for potential users. Geneva: World Health Organization. WHO (2004) Maternal mortality in 2000: estimates developed by WHO, UNICEF and UNFPA. Geneva: World Health Organization. Other relevant references on measuring maternal mortality Danel I, Graham W, Stupp P, Castillo P (1996) Applying the sisterhood method for estimating maternal mortality to a health facility-based sample: a comparison with results from a household-based sample. International Journal of Epidemiology, 25 (5), 1017–22. Graham WJ, Fitzmaurice AE, Bell JS, Cairns JA (2004) The familial technique for linking maternal death with poverty. Lancet, 363 (9402), 23–7. Hill K, El Arifeen S, Koenig M, Al-Sabir A, Jamil K, Raggers H (2006) How should we measure maternal mortality in the developing world? A comparison of household deaths and sibling history approaches. Bull World Health Organ,84 (3), 173–80. Macfarlane A (2001) Enquiries into maternal deaths during the 20th century. In: Why mothers die 1997–1999. The Confidential Enquiries into Maternal Deaths in the United Kingdom. London: RCOG Press, pp. 346–57. Olsen BE, Hinderaker SG, Lie RT, Bergsjo P, Gasheka P, Kvale G (2002) Maternal mortality in northern rural Tanzania: Assessing the completeness of various information sources. Acta Obstetricia et Gynecologica Scandinavica, 81 (4), 301–7. WHO (2004) Beyond the numbers. Reviewing maternal deaths and complications to make pregnancy safer. Geneva: World Health Organization. UNFPA (2002) Distance-based learning course. Reducing maternal deaths: selecting priorities, tracking progress. http://www.unfpa.org/dlpi/courses.htm. Data analysis plan for survey to measure the maternal mortality ratio for ‘Myland; using SSS Introduction Background information Data entry Data cleaning Data quality checks Data analysis Data archiving

View the publication