M ISRAR KHAN
ISLAMABAD (April 20 2006): The Asian Development Bank (ADB) has pinpointed that the frequently changing data collection methodology of the Federal Bureau of Statistics (FBS) during the last decade has affected the reliability and comparability of its survey data.
The bank has proposed adoption of reliable and independent internal mechanisms for supervision and validation of fieldwork to address the problem.
For instance, different data sources that are not necessarily comparable in terms of sample design, seasonality, or methodology, are often used to examine poverty trends, making the data unreliable.
The Household Integrated Economic Survey (HIES) is the main source of data for poverty estimates in Pakistan, and should be strengthened, at least to the extent of producing regular, credible data, even at intervals of two or three years. This would serve the purpose of poverty monitoring effectively.
The ADB in its Pakistan Poverty Assessment Update titled "The Reliability and Credibility of Statistical Data for Poverty Analysis in Pakistan" said, "The reliability and credibility of Pakistan's poverty database, which is generated primarily through household surveys, has been debated for a long time. Issues of concern include updating of the sampling frame; survey comparability; availability when needed; frequent changes in field methodology; and the quality of questionnaires used."
Criticising the Pakistan Social and Living Standards Measurement (PSLM) survey, the bank said that it does not, however, adequately monitor all millennium development goals (MDGs) and Poverty Reduction and Strategy Paper (PRSP) indicators. Fertility and child mortality rates, for instance, cannot be estimated through this survey.
One important MDGs indicator is the "proportion of children who reach Grade 5 of those admitted in Grade 1. This indicator requires data on school dropouts, which the PSLM does not provide. Like the PIHS, the PSLM survey also fails to monitor several other MDGs/PRSP indicators, such as maternal mortality and malnutrition (underweight children). The new survey is, therefore, inadequate for the purposes of monitoring all the MDGs/PRSP indicators.
The Bank also said that the credibility of data depends largely on how independent the executing agency is, and, more importantly, on the quality of the data itself.
The FBS ensures data quality by carrying out consistency checks and 'cleaning', but validating survey data in the field is the key to enhancing data quality. This is usually done through a post-enumeration survey (PES), which is not common in Pakistan. An alternative would be to validate FBS fieldwork externally, although this is not a common practice among data generating agencies in the developing world.
Third-party validation may in fact weaken the FBS's fieldwork supervision ability. Rather than introducing third-party validation to verify FBS household-level data, reliable and independent internal mechanisms for fieldwork supervision and validation could be developed to address the problem, it added.
It is important that such a mechanism establishes its credibility through long-term improvement in its data collection techniques. This will boost the confidence of the local staff and strengthen the FBS's ability to carry out surveys.
The Bank's background Paper-2 assesses the reliability and credibility of the household surveys that generate data for poverty analysis in Pakistan. It focuses on the data generated by four household surveys: the Household Integrated Economic Survey (HIES), Pakistan Integrated Household Survey (PIHS), Pakistan Demographic Survey (PDS), and Labour Force Survey (LFS), all the four are carried out by the Federal Bureau of Statistics (FBS).
A common observation on large surveys such as the HIES and PIHS is that the income accruing to the highest income group is seriously understated, and that the poorest are inadequately represented. This issue of the representation of high-income groups surfaced more seriously in the 2001-02 PIHS/HIES, where low- or middle-income group primary sampling units (PSUs) were enumerated against high-income PSUs.
The FBS report that was published subsequently blamed its field supervisors for the negligence. This is a poor defence - this is a serious sampling task and should be managed regularly well before a survey is started, it added.
The FBS has initiated a new survey, the Pakistan Social and Living Standards Measurement (PSLM) survey, but it does not contain a module on birth history to estimate fertility and child mortality rates. The PDS remains the main data source for these indicators.
Although a reliable data source, the PDS also faces certain concerns: recent changes in methodology may have affected fertility and mortality rates, but this impact has not been evaluated. In addition, data on children's height and weight is not regularly available to monitor child malnutrition. This information could be obtained through the PDS if female enumerators were inducted into the survey. It would be worth investing in the PDS for reliable demographic and health data.
The Bank said that in general, labour data remains inadequate; it would be worth investing more in the Labour Force Survey (LFS) to make it an annual survey.
The ADB also said that there is considerable debate on the urban sampling frame used by the FBS. Using the 1998 population and housing census data, the FBS revised its rural sampling frame, which was then used in the two combined rounds of the PIHS/HIES.
However, the FBS's urban frame was last updated in 1995, and this might affect the rural and urban distribution of a sample. It also affects the overall estimates of poverty, as well as comparability across years, and, thus, needs to be updated urgently. Frequent changes in data collection methodology also affect the reliability and comparability of survey data. Such changes include changes in the reference period for reporting births and deaths (the PDS) and for the expenditure module of the HIES/PIHS. Such changes may be required to improve data quality, but the frequency of changes in reference period may affect reporting quality and data comparability across surveys.
The quality of data produced by the household surveys so far implies that the quality of training and supervision of surveyors needs to be improved. One reason for the recent controversy over household size, which is an important variable in estimating poverty, was that ill-trained enumerators did not enlist household members properly.
The Bank said that the supervision system should be changed to allow supervisors to manage their field teams for the entire survey period, as was done by the FBS in the PIHS rounds. Household survey monitoring teams consisting of regular field staff posted at national, regional and field offices across the country could also monitor surveys more effectively.
FBS supervisors are responsible for many things: overseeing the identification of enumeration blocks (PSUs); household listing; selecting households using systematic random sampling techniques; field enumeration; editing and cross-checking data entries; and finally, "cleaning up" the data collected, the bank adds. The supervisors cannot carry out all these assignments efficiently, and should be relieved of some of the duties.
For example, when the 2001-02 PIHS data was cross-checked, some low-income households were found to have been included in high-income areas. The FBS blamed its field supervisors for this negligence. However, this is a serious sampling task and should not simply be relegated to the supervisors. Household survey monitoring teams consisting of regular field staff posted at national, regional and field offices across the country could be used effectively to monitor surveys.
The concept of teamwork is largely absent in the FBS's fieldwork strategy. One reason that non-government agencies produce better-quality household survey data is that their fieldwork teams are headed by supervisors who are based in the field for the entire duration of the survey.
While on the other hand, the FBS module is different: it uses mostly male enumerators who are not accompanied by the supervisors on a daily basis. The PIHS series is an exception; data was collected by mobile teams of male and female enumerators who were managed by a team of supervisors on a daily basis. It may be costly for the FBS to spare its staff, as supervisors who go into the field every day with their team, but such improvement in the field strategy are necessary to improve the quality of household survey data.