The age of data

Regardless of personal views, it's evident that the bureaucracy and political leadership are fundamental in designing and implementing government policies and hold considerable expertise.

This is why it is crucial to explore how this process can be strengthened in today's era of digital abundance and data overload. With an exponential increase in data and digital technologies, and the transformative impact of AI, relying on traditional methods like the opinion of an expert or her/his gut instinct for policymaking has become impractical.

Using complex information for decision-making now demands more sophisticated approaches. Furthermore, with social media widespread across all levels of society, people expect more from their governments. Governments, on the other hand, have limited resources which demand that they use these resources as efficiently as possible to tackle development and other challenges.

Evidence-based policymaking needs more than just lip service to have effective and meaningful discussions in policy. Data, like any other resource, holds significant value but only if it is effectively utilized. Without proper use, it's just like any untapped resource in your backyard – potentially valuable, but practically useless.

The private sector has been effectively utilizing the potential of data in its business decisions. The public sector has so far been stuck in the pre-digital era policymaking mechanisms and has limited exposure to developing government analytics for policymaking.

I understand that the public sector operates under distinct dynamics compared to the private sector, despite the significant benefits that data has brought to the latter. It is not the case that everything is measurable or that everything measurable is of great importance for public policy.

Also, it is not data – but data that matters. Easy-to-measure indicators may lead to false alarms leaving real issues unattended. It is also needed to protect space for judgment, discretion, and deliberation because not everything that matters can be measured. Data analytics in the public sector should be rooted in the pursuit of a brighter future, where one prioritizes not only acquiring the right data but also utilizing its potential to enhance public policymaking.

Policymakers should realize an urgent need for data analytics across various sources such as administrative records, public procurement data, health and education statistics, and public sector development projects. For example, in the realm of justice, data can be used to pinpoint inefficiencies within the judicial process and suggest necessary reforms. The health and education sectors have large prospects for improvement using data and AI to address even the needs of individuals. Similarly, in public procurement, utilizing data to develop clear and effective rules for procurement officers is essential to ensure that policies lead to the desired outcomes.

Besides structured data, it is important to understand the significance of unstructured analysis of policy documents, regulations, and tenders using techniques such as sentiment analysis, topic modelling, and machine learning. This comprehensive approach allows one to uncover hidden patterns, identify policy focus areas, and enhance decision-making processes.

One significant challenge that Pakistan, along with many other developing nations, encountered from the beginning of the Sustainable Development Goals (SDGs) programme till today is the lack of data availability for many indicators at both national and sub-national levels. Without data on an indicator(s), it is not possible to track progress on it. This underscores the urgent need for investment in data management, data platforms, and capacity building to harness data for actionable insights. There is an urgency for an investment in developing a data ecosystem that can provide comprehensive, accurate, disaggregated, and timely data. Currently, unstructured data constitutes about 80 per cent of all data generated, yet efforts have largely been focused only on handling traditional survey data and other standard datasets.

By effectively leveraging data resources – including management, common platforms, data science, and machine learning – and combining them with the deep insights of domain experts such as bureaucrats, technocrats, and political leaders, we can make better policy decisions, allocate resources more efficiently, and ensure that interventions are precisely tailored.

In conclusion, enhancing the government's decision-making process at every level through the integration of data analytics in policy formulation and execution is crucial. Data analytics should not replace expert opinions but serve as a vital tool that enriches policy discussions, enabling the identification of weaknesses and the facilitation of targeted improvements across various sectors.

Additionally, analytics should augment the quality of discussions concerning public administration improvements, rather than simply dictating actions based on specific outcomes. This approach is essential for making informed decisions, managing projects efficiently, and effectively monitoring and evaluating various initiatives, ultimately leading to better governance and more effective public service.

The writer is a professor of economics at Quaid-i-Azam University, Islamabad.