AI journey: fueled by data, not hype

Advanced technology is the driving force behind modern states. Any state or society that lags in technology and innovation risks being left behind. Among the latest innovations, artificial intelligence (AI) stands out as a transformative force impacting all sectors. Unlike missile or nuclear weapon technologies, which require specialized expertise, materials, and restricted knowledge, AI is scalable and accessible to those who build the capacity to use it.

The IMF recently ranked Pakistan 117th out of 174 countries in AI readiness. One of the critical components for leveraging AI effectively is having the right kind of data. In the era of rapid technological advancement, two essential requirements are data and the capacity to make sense of it. Data has become the cornerstone of artificial intelligence (AI), large language models (LLMs), natural language processing (NLP), machine learning (ML), and a myriad of other transformative tools. How data is utilized, prepared, and consumed significantly impacts the capabilities of individuals, enterprises, and governments.

Despite AI's vast potential and wide-ranging applications, Pakistan seems to not benefit from it a lot mainly because it does not have sufficient data infrastructure which is the basic fuel for AI. For Pakistan's public sector, the data boom presents daunting challenges alongside unique opportunities. To convert these challenges into opportunities, investment in data infrastructure and the capacity to handle both structured and unstructured data is essential.

Think of AI as a powerful engine capable of incredible feats, but only with high-quality fuel. In this case, the fuel is data. The more data we have and the better its quality, the more effective AI becomes. This applies to everything from large language models to machine learning algorithms that power various applications.

In Pakistan, over time, the digitization of thousands of documents has led to a proliferation of unstructured data, creating an urgent need for organizations to prioritize this process. The digitization drive has generated a massive amount of unstructured data. But unlike structured data in spreadsheets, this unstructured data is messy and requires specialized tools to analyze for which the public sector has no capacity.

This unstructured data is a goldmine for informed policymaking. Imagine a world where policymakers can instantly access and analyze data on any issue, or parliamentarians can efficiently gather information on past policies. However, unlocking this potential requires more than just fancy software. We need to build internal expertise in data analysis. Relying solely on external consultants isn't sustainable. We need to create a culture shift where data analysis is seen not as a chore but as a critical tool for effective governance.

The public sector faces significant challenges in improving its policies through data-driven, evidence-based approaches due to the complexity of handling unstructured data. Historically, data training has focused on structured data stored in predefined formats, which organizations have managed effectively for decades. Unstructured data, often referred to as the ‘white whale’ of both private and public sectors, represents a vast reservoir of untapped potential. The public sector possesses a wealth of this data, yet extracting value from it remains exceedingly complex.

If a senior leader, public official, or policy analyst asks staff to compile everything known about a particular issue, they should be able to mine the entire database and quickly pull out abstracts, data, information, or policy recommendations. Similarly, if a parliamentarian asks an aide to gather information on policies from the past three decades, they should be able to do so efficiently after the necessary training. Unstructured data is a goldmine that, when properly leveraged, can significantly enhance policymaking.

With ongoing developments in AI and LLMs, unstructured data is growing exponentially, and this trend shows no signs of slowing down. It is time for the public sector to build scalable, strategic capabilities. The time is now for partnering and building teams to harness the power of unstructured data. Imagine having hundreds of policies and other documents at your disposal; unstructured data can become a major part of this resource. However, accessing and understanding unstructured data is challenging because it is not stored in the same manner as structured data, like spreadsheets and tables.

There is a dilemma. Given the limited supply of skills within the public sector, relying solely on external expertise is not a viable first option. Local expertise is essential to fully benefit from external capacities. However, building local capacity also requires time and targeted skills.

The primary challenge for the public sector is the lack of awareness about how data can exceed the expectations of senior leadership and bureaucracy. This untapped potential is hindered by the absence of clear, standardized success metrics. As the saying goes, ‘you need to crawl before you walk’. It is crucial to start with a single sector, develop and execute it, and then expand ML/LLM models to other areas.

For public policy officials, understanding that rigorous, quantitative research and education are the best guides for effective public policy is essential. Embracing the power of unstructured data through NLP and AI can revolutionize the way policies are crafted, evaluated, and implemented, ultimately leading to more informed and impactful governance in Pakistan.

The road ahead won't be easy. Building expertise takes time, and we can't jump straight into the deep end. A good first step is to focus on a single sector, develop successful AI models, and then gradually expand. This ‘crawl before you walk’ approach ensures a solid foundation for future growth.

Investing in data and AI isn't just about chasing the latest tech trends. It's about building a better future for Pakistan. By embracing data-driven approaches and empowering ourselves with the latest developments in the field, we can revolutionize policymaking, leading to more informed decisions and impactful governance.

The choice is clear: Pakistan can either watch the AI revolution pass us by or we can become active participants. The key lies not in blind faith in technology, but in a strategic investment in data – the fuel that will propel our journey into the exciting world of AI.

The writer is a professor of economics at Quaid-i-Azam University, Islamabad.