Pakistan’s dismal performance on education-related Millennium Development Goals (MDGs) (now replaced with Sustainable Development Goals -- SDGs) is a cause of concern for governments, civil society organisations and international donors. Lack of political ownership to improve education-related indicators in Pakistan was one among many reasons.

Alif Ailaan, with the help of UK-Aid, came up with an innovative campaign idea in 2013 to reach out to the political class and governments at different levels in order to increase awareness about the education crisis in Pakistan. Doing research collaboration with SDPI and taking financial help from the UK Department for International Development, the campaign seeks to put education at the centre of public discourse in Pakistan.

To state the obvious, Pakistan’s education sector is in a bad shape. It needs all the help it can get. The intention is not to nitpick over the efforts of the few organisations that are working to improve the state of education in the country. Having said that, since resources are scarce, whether it is money coming from the Pakistani government or from the global pool of development expenditure, we should not be shy about voicing concerns about an organisation’s lack of clarity when it is going about its work, whether it is feet-on-the-ground infrastructure development, research or advocacy that is the organisation’s area.

On May 24, 2016, Alif Ailaan launched its 2016 district education rankings report which is the fourth report it publishes each year since 2013. Alif Ailaan-SDPI District Education Rankings aim to achieve the following five objectives:

To produce a measure of education standards in Pakistan that covers important policy areas for which data is available, including access, retention, learning outcomes, gender parity and school facilities and infrastructure;

To establish a basis for comparison, across time and administrative units of different parts of the country, to track education sector performance, especially in government schools;

To instigate debate and competition in the national political discourse across districts and provinces (and/or regions);

To increase the use of evidence, especially official government data, in assessing the state of education and promotion of evidence-based decision-making in education; and

To make robust policy recommendations for improving the national data regime for education statistics.

This article will critically evaluate Alif Ailaan’s stated objectives against its own published reports and will point out flaws in the overall design of the campaign. The purpose here is to engage education advocacy campaigners in a constructive debate to further the important cause of education in Pakistan. A brief description of Alif Ailaan’s data sources is necessary before discussing the main argument of this article.

Alif Ailaan-SDPI district education rankings reports (and its other reports) were developed mainly by using three national data sets on primary and secondary education in Pakistan. Two of these data sets come from official government sources while one data source is by volunteers and non-profit organisations led by Idara-e-Taleem-o-Agahi (ITA).

The first official data source used in Alif Ailaan’s report is the National Education Management Information System (NEMIS) which in turn is developed by combining the provincial/regional Education Management Information Systems (EMIS). Provincial/regional EMIS produce Annual Statistical Reports -- the outcome of Annual Schools Census (ASC) -- and all provinces/regions have been regularly collecting data on yearly basis since 1992. The data is mainly about education service delivery (access, facilities, teachers, rooms etc) and various indicators of enrolment. Unfortunately, these Annual Statistical Reports are published with a lag of two years, which with little effort, can be reduced to same year.

The second official data set used in Alif Ailaan’s reports is data from the Pakistan Social & Living Standards Measurement (PSLM) surveys conducted by Pakistan Bureau of Statistics (PBS). PBS has been conducting these PSLM surveys regularly since 2004-05. These population-based household surveys are conducted alternatively at the provincial and district level each year. Among other social indicators, the data covers a multitude of indicators on education which together with Household Integrated Economic Survey (HIES) provides linkages of socio-economic characteristics of households to the education attainments in the households. Like Annual School Census data sets, PSLM data sets too come with a lag of two years.

The third data set is Annual Status of Education Report (ASER) which mainly covers the student learning outcomes in both government and non-government schools. The data is collected by volunteers and non-profit organisations led by Idara-e-Taleem-o-Agahi (ITA) since 2008. ASER data set for a year is normally available in the month of January next year without any lag period.

To gauge the state of education across regions/provinces of Pakistan, Alif Ailaan’s district education rankings reports develop two composite indices based on the above three data sets. Its first index, Education Score Index, is made up of four components namely access, retention, learning and gender parity, all with equal weights. The second index, School Infrastructure Index, is made up of five components of availability of electricity, water, toilets, boundary walls and condition of the school buildings. While much can be said about the robustness (or lack of it) of these composite indices to measure what it want to measure, the following points explains how Alif Ailaan’s campaign fails to achieve its stated objectives.

First major problem in Alif Ailaan Annual District Rankings Reports is presenting two-year old data as fresh data for the year of the published report. Alif Ailaan uses two-year old data sets to calculate its two main indices for ranking of districts/regions. Thus, the reports which were published in 2013, 2014 and 2015 is the performance evaluation of the previous provincial governments while the report published in 2016 can be seen as performance evaluation of the current provincial governments.

While Alif Ailaan reports did mention this limitation of the data in its methodology section but tables/graphs/figures in the reports were titled with the year of publication of Alif Ailaan reports. This is against the international convention of data presentations in reports and thus led to confusion. For example, Pakistan Tehreek-e-Insaf leaders shared the findings of 2014 and 2015 reports widely on social media claiming success of the PTI policies in Khyber Pakhtunkhwa. The PTI chief, Imran Khan & KP Education Minister Mr Atif Khan tweeted in May 2015 that KP education score has improved over previous years by depicting ‘an improvement in KP education’ which is actually based on the data of ANP-PPP era. On top of that, both these tweets were retweeted by Alif Ailaan’s official twitter handle.

As can be seen, this is problematic on many counts. Alif Ailaan political campaign’s main objective is ‘to instigate debate and competition in the political discourse’ on the state of education in Pakistan. However, their District Rankings Reports did not instigate the right kind of political competition between different political parties and regimes. Performance improvement in KP as a result of policies of the previous regimes was shown as an improvement brought by the current regime.

The current KP government was misled by Alif Ailaan, as the provincial government may now think that it is due to PTI’s policies that education scores improved in the province. This could have been avoided if international best practice for data reporting were adopted by Alif Ailaan. It is surprising that Alif Ailaan also promotes tweets done by PTI officials while campaign managers at Alif Ailaan do know and accept that data used in these reports pertains to previous regimes.

The second major problem with Alif Ailaan’s political campaign is that their reports and social media campaign put the burden of accountability regarding primary and secondary education improvement on MNAs and MPAs. There is a separate section in their annual District Rankings Report on ‘holding our elected representatives accountable’ where a list of prominent MNAs has been given with the education rank of the district from where the MNA has been elected. On top of that, Alif Ailaan published a separate Mid-Term Report Card in November 2015 where it ranked every MNA for its education score of his/her constituency. MNAs are constitutionally not accountable for primary and secondary education as it is a provincial subject.

The third problem with Alif Ailaan’s report and campaign is the construction of the composite indices of education scores which are the basis of their district/region rankings. The three data sets used in Alif Ailaan District Ranking reports cover different dimensions of the education sector in Pakistan. Instead of lumping them together, it will be better to keep these reports separate as all reports cater to different policy needs. Important information is being lost during the process which can otherwise guide public policy on education improvement in Pakistan.

The PSLM dataset, for example, links household socio-economic characteristics to the education attainment within a household. Researchers and advocacy campaigners can use PSLM data sets to advise the government on policy measures which, for example, affect the demand side dynamics at the household level. Similarly, the provincial school census reports combined in NEMIS data set mainly pertains to the access indicators of schools, classrooms, teachers, facilities and infrastructure and thus caters for the supply side of the education sector. The ATA data sets mainly cater to the quality side of the education and thus can guide government policy on improving student learning outcomes. Lumping these completely different data sets in terms of its policy implications and of different time period may only lead to a confused policy advice.

The fourth and the last point is the lack of robustness of the composite indices with which Alif Ailaan ranked districts and regions/provinces. School infrastructure index, which is one of two main measures adopted by Alif Ailaan, is an indicator to show the degree with which government schools are equipped to serve the communities and children they are meant for. Specifically, it is made up of 5 components mainly; availability of water, electricity, toilet, boundary wall and building condition of the school. A district with fewer schools per population but equipped with the above facilities will rank higher than another district with more schools per population but not fully equipped.

While school density is an important indicator of school infrastructure as it determines access to school, Alif Ailaan’s rankings don’t consider it worth including. This is despite the fact that Annual School Census data is very rich which can be used to construct a more robust indicator for school infrastructure.

While it is commendable that the Alif Ailaan campaign brought in the debate around primary and secondary education in print, electronic and social media; it is saddening that they are not doing it the right way while they have the capacity. The most beneficial way to use these rich data sets for advocacy is to present it separately to policy makers in different seminars and workshops. In fact, the government organisations which generate these data sets are able enough to present it to provincial education ministers and politicians.

With the help and support of organisations like Alif Ailaan, the same government organisations can better disseminate their data and reports which will be helpful in better policy advices to the policymakers.

The author is a freelance development consultant