Navigating the Adoption of AI for Policymaking

Introduction

This paper follows from a seminar presentation that I gave at the Geary Institute for Public Policy in University College Dublin titled Beyond the Hype: Promises, Pitfalls, and Illusions of AI in Policymaking. In that presentation, I made the case for the need of capacity building in government to navigate the current hype on the adoption of artificial intelligence (AI) tools. In this piece, I rationalise my experiences over the last decade from working with various types of public organisations across different countries in the intersection of economic policymaking and computational methods. The reader may find the tone of this piece critical and even provoking, nevertheless not without a purpose, that of creating awareness and critical thinking among policymakers considering the adoption of AI in their organisations.

Currently, and especially in the Anglosphere, a big push towards the integration of AI in various aspects of public administration and public policy is evident. While the term AI is loosely used in media and government documents, it tends to refer to what Russel & Norvig (2002) call the acting humanly type of AI. Under this AI paradigm, theories explaining why humans behave the way they do is of secondary importance. Instead, discovering data patterns is prioritised, privileging predictive accuracy over explanatory mechanisms. Hence, the colloquial use of AI often refers to machine learning frameworks, such as large language models (LLMs), which are designed to analyse large-scale complex data and are often seen as black boxes. As I write this paper, it is easy to find many examples of this hype published in mass media, from ministers and politicians talking about AI to departments recruiting experts on large-language models; from memorandums of understanding (MOUs) between AI companies to multi-million contracts granted to big tech; from government-sponsored conferences on the use of AI in policymaking to the creation of AI strategy delivery leads in various ministries.

There are indeed potential benefits stemming from the adoption of AI, but there are also overblown expectations that make it difficult to discern between realistic potential outcomes and wishful thinking. These sorts of hype in the public sphere are not new, especially in the Global North. For example, in the 1970s, the publication of The Limits to Growth (Meadows et al., 1972), triggered the development of System Dynamics models across governments and international organisations. More recently, we have seen governments hyped around tools/frameworks such as nudge (circa 2010-15), big data (circa 2012-16), block chain (circa 2014-18), and data science (circa 2015-23), all of them materialising in government expenditure (e.g., recruitment, creating intelligence units, outsourcing companies, creating datasets). Hypes can be good when they generate momentum and trigger the adoptions that otherwise would not happen due to excessive scepticism in certain technical analysis guilds. For example, the data science hype paved the way for economists in some central banks to consider text analysis seriously (Kalamara et al., 2022). However, overspending due to inflated expectations is one of the obvious drawbacks of hype-driven adoptions. A failed adoption may just end up in a graveyard of abandoned/depreciated tools that became too difficult/costly to deploy, a sort of digital elephant. Navigating a landscape shaped by promises and mined with pitfalls is challenging, especially when technical capacity is limited and there is a misalignment of incentives between civil servants and their organisation. Consequently, amid a hyped environment, it is easy to get enchanted by the illusions of AI.

There is something different about the AI hype

The current AI hype has distinctive features that were unseen in previously similar hypes. First, the degree with which contemporary AI tools such as LLMs can be generalised in the public sphere is unprecedented. Previously adopted analytic tools such as System Dynamics, Microsimulation, Dynamic Stochastic General Equilibrium (DSGE) Models, and Agent-Based Models (ABMs), have traditionally remained in specific governmental organisations with specialised technical teams, for example, public health authorities, finance ministries, central banks, and urban development offices, respectively. In contrast, current AI tools can be deployed across a wide set of tasks that span several (if not all) governmental units, for example, summarising and translating documents, searching the web for studies providing policy evidence, extracting knowledge from large legal corpora, coding a simple simulation or statistical model, producing short-term forecasts, cleaning and classifying data, evaluating applications in procurement processes, and supporting public-facing activities such as answering queries (via chat bots) and producing communication materials. Hence, the potential degree of penetration that contemporary AI has is unparallel.

The second feature that makes this hype unique is how complicated this technology is. While already adopted analytic frameworks such as DSGEs and ABMs are far from being simplistic tools, their canonical assumptions, underlying theories, sensitivities, limitations, estimation methods, validation strategies, and data needs are well understood by communities with both technical and domain expertise. Most of the times, their deployment is framed around specific questions associated with causal mechanisms or theories of change. The structure or architecture of DSGEs and ABMs is–to some degree–the reflection of hypothetical causal chains. This allows for a certain degree of flexibility to adapt these tools to specific policy questions. Furthermore, should a scenario, evaluation, or prediction fail, policymakers can go back and revise the model and its underlying theory to understand what went wrong. With contemporary AI, it is considerably more difficult to gain this kind of insights and flexibility. For example, the sheer architecture of deep neural networks combined with the self-attention modules that underlie LLMs is so complicated that it is impossible to map it into any relevant socioeconomic theory. While the black-box nature of these tools may be acceptable when supporting administrative tasks, it may amplify risks and biases in decision-making that involves interventions, as it is hardly possible to audit these tools if an intervention fails. For instance, contemporary LLMs can be so large and complicated that the concept of open-source code as a practice for ensuring  transparency has lost relevance (as it is extremely difficult for individuals to audit these models), so the notion of a fully open model, which encompasses open-source code + open parameters + open training data, has become the gold standard to promote transparency and reproducibility.

Third, once it has been set up, an AI tool can be adopted relatively quickly. The wide range of tasks enabled by foundational AI models have created economic opportunities for companies adapting these tools to domain-specific tasks. The growth of the AI industry has given space for these tools to mature enough to become highly user friendly, so much that little technical knowledge (if any) is required to use them. Such ease of use stands in stark contrast with previously mentioned analytic frameworks where substantial investments in human capital are needed just to be able to use them. Of course, the more specialised the application of an AI tool becomes, there more technical knowledge is required. However, even in highly specialised applications, the bar for using readymade AI tools is significantly lower than for other analytic tools used in policymaking.

Fourth, as complicated tools that are easy to use in many types of tasks but difficult to develop for a single person or public body, a large portion of industry-ready AI tools have been developed by private companies. These companies may act as gatekeepers if no reliably publicly available alternative exists. Not only is this an issue of access, but also of extreme reliance on outsourcing services, which can put governments at the mercy of incomplete contracts, short-termism, and lobbying pressures. This situation stands in stark contrast with the adoption of previous analytic frameworks where government departments would invest in technical teams, international organisations would provide technical assistance, academics would engage in knowledge transfer, and private consultants would provide supplementary analytic services instead of replacing internal capacity. Those adoptions would rarely lead to gatekeeping dynamics and situations of long-term over-reliance, unless highly specialised physical technologies would be involved (Srivathsan et al., 2024). The monetary size of the contracts one sees today when the adoption of AI is announced by far surpass conventional technical consultancy services (e.g., according to OECD (2025), USD 3.3 billion in federal AI-related contracts in 2022). Furthermore, these deals take place in a global economic environment in which the same technological firms are gradually capturing more of the computational resources needed to deploy their and other types of analytic tools (i.e., the production and ownership of chips and memory) (Larson et al., 2024).

A slippery slope

The challenge of successfully navigating the hype for using AI for policymaking adoptions is a difficult one. Obstacles related to institutional design and incentive structures need to be circumvented. When the incentives of civil servants and policymakers (e.g., career progression) do not align with the policy goals of an organisation (e.g., poverty reduction), inefficiencies, private gains, and perverse incentives may arise and prevail, leading to policy incoherence (Guerrero & Castañeda 2021, 2024) and, in the case of adopting new analytic tools, digital elephants. Notice that a misalignment of incentives is not necessarily the result of bad intentions or a flawed institutional design. However, in an environment distorted by hype and overblown expectations, even the most resilient organisations may lose their way in the mist of promises and illusions.

When expectations about a new method are inflated, their adoption may seem like a strategic priority for the leadership of a public entity, crowding out priorities on long-standing more fundamental problems. To illustrate this, allow me to discuss the case of a hypothetical country that is going through a housing crisis. In this country, properties have become unaffordable for the median worker and rents eat up most of their income. This housing crisis has been going on for two decades, and a new government has been elected under the promise of remedying this problem. Effectively, by electing a new government, society has given it the mandate of putting an end to the housing crisis.

With the aim of designing policies to tackle the housing crisis, the government consults experts and receives two alternative explanations for the housing crisis. The first explanation suggests that this is a problem of aggregate supply and demand. Hence, the solution is to build more homes to push supply and reduce prices. The second explanation is about who is buying what (or how many) properties, and how much. The first explanation suggests interventions that incentivise or help developers to build more and faster. The second looks at the housing crisis as a cross-sectional problem where market dynamics may lead certain actors to concentrate real estate in ways that housing becomes unaffordable. It involves more nuanced and perhaps politically uncomfortable interventions such as restrictions on who can purchase what, social housing, rental caps, changes to the stamp duty, restructuring council tax, etc.

Now, suppose that time, monetary, logistical, and political resources are scarce, so the government must choose one of the two explanations to focus its policy efforts on. Suppose that a large tech firm offers the government a large language model to slash planning delays by automating parts of the approval process of new developments. The promise is that this LLM could accelerate building 1.5 million homes during the government’s term. In a hyped environment, bringing AI into government seems ideal to the civil servants. Not only does it apparently align with ending the housing crisis, but it also elevates the profile of the civil servants in terms of showing vision and leadership.

The government opts for the first explanation, without giving too much consideration to the second one. An LLM is developed, deployed, and maintained by the tech firm under a hefty multi-year contract. Towards the end of the administration, modest time savings in the planning process are achieved and nearly 1.5 million homes are built. However, to the government’s surprise, housing prices remain at an all-time high level. Housing scholars later find that, while more houses were built, banks, investment funds, pension funds, and private equity firms were systematically investing in the housing market, buying entire developments (for renting) as soon as they are built. As the 19^th century French economist Jean-Baptiste Say would put it: the housing supply created its own demand, nullifying the policy’s impact on prices. The problem was clearly more nuanced than one of aggerate supply and demand. However, driven by an AI hype and incentives to display leadership, the civil servants did not consider the second explanation seriously. In this hypothetical scenario, the government not only failed at its mandated task, but also spent resources on a digital elephant that generated marginal benefits.

What the previous hypothetical scenario portrays is how the convergence of short-termism, illusions fed by inflated expectations, misaligned incentives, and a hype for AI can make institutions and civil servants fall prey to illusions. Interestingly, this situation may be free of political interests or bad intentions, leading the best-intended civil servants and politicians to create voluptuous AI projects with marginal gains or even complete failure. It shows that, even when having the best intentions, there are structural elements that may easily lead to failed adoptions.

Navigating the adoption of AI

If navigating through promises, pitfalls, and illusions is difficult, how can we gain clarity to smooth the ride? There are no universal answers as technologies, problems, organisations, and technical capacities vary across contexts and are constantly evolving. Nevertheless, from my own experience, I have identified useful guiding questions that civil servants should ask when considering the adoption of any type of methodological innovation. These questions aim at mitigating hypes and anchoring a potential adoption to the problem and institutional context at hand.

Context questions:

• What is the big-picture problem that my organisation must solve?
• What policy instruments does my organisation have?
• How are policy instruments informed or shaped by evidence?
• What is the nature of the main task (e.g., administrative, evaluation, forecasting, measurement, scenario analysis, counterfactuals)?
• What knowledge do I possess about the system that I wish to intervene (e.g., theories, causal chains, driving factors)?
• How directly would the potential adoption reshape my organisation’s policy instruments?

Data questions:

• What data does my organisation have?
• What kind/size of data was this method designed to use to perform best?
• Is it necessary/feasible to acquire new data?
• If the adopted method is pre-trained, would the training data be available?
• If licensing or collecting data is necessary, is this sustainable in the long run?
• Would data collection/generation be robust to license disruptions or to the closure of private providers?

Computing questions:

• Using my organisation’s computational resources, how long does it take to train and fine-tune the potential adoption?
• How much memory does it need?
• How frequently will the adopted tool need to be fine-tuned and retrained?
• Is retraining computationally feasible with the frequency desired by my organisation?
• What is the loss of performance if retraining is performed infrequently?
• What type of hardware is ideal to run the potential adoption?
• What are the costs (financial and computational) and benefits of acquiring specialised hardware?

Opportunity cost questions:

• What alternative methods are designed to analyse the problem at hand?
• What alternative methods are used in similar tasks?
• What type of method(s) would be the most relevant/adequate for the data at hand?
• Are there alternative methods that offer performance close to the considered one but at a fraction of the cost (computational and otherwise)?
• Would it be possible to consider more than one tool and produce consensus evidence?
• Are there complementarities between the proposed adoption and incumbent analytical tools in my organisation?
• Do I expect that the adoption will systematically outperform alternative tools as new data becomes available in the future?

Internal capacity questions:

• Does my organisation possess technical capacity to build and deploy this tool in-house?
• What are the potential benefits and costs of training a team in-house?
• Are there technical teams in other government departments who could facilitate knowledge transfer?
• Can we exploit economies of scale from prior adoptions by other public organisations?
• What are the potential benefits and costs of outsourcing external computing services?
• Would outsourcing external computing services risk data confidentiality or violate existing legal frameworks?
• How challenging would it be to create a contract that prevents moral hazard from third-party providers?
• How would moral hazard and unexpected price increases impact the sustainability of this adoption?
• What would be the risks of a data leakage from a third-party contracted cloud service?

Asking the previous questions can provide some guidance and help contextualise the true potential of adopting AI. Nevertheless, it is also useful to try to think of an adoption in terms of how far it can reach within or beyond our own organisations. This is something that Helen Margetts and I thought for some time, leading to the 3Ps framework (Guerrero & Margetts, 2024). The 3Ps stand for three dimensions that characterise the adoption of analytic tools:

• Pace: Can some methods be adopted and deployed at a faster pace than others?
• Penetration: Can some tools achieve a deeper degree of penetration in public organisations in terms of how generalised is their adoption?
• Profundity: Can adoption translate into fundamental changes in the policymakers’ understanding of the system they are intervening and everyday practices?

A policymaker may prioritise one or more of these dimensions. This means that, just because an AI tool does not produce a profound change in policymaking practices, it does not mean that it is less desirable. Deploying a tool that automates certain tasks without making us rethink the workings of a social system is a perfectly legitimate priority as long as the true benefits are not inflated and they align with the organisational mandate. What is important to recognise, nevertheless, is that policymakers should be aware of the distinction between influencing policymaking and improving administrative procedures. Often, grandiose claims are made about tools that automate administrative tasks though hyperboles about great societal impacts from cutting down times in administrative procedures.

When thinking in terms of the 3Ps, we can represent their space as shown in Figure 1. This is an example of how one could place the application of a specific tool along the three dimensions of its adoption. In this example, a large language model to automate administrative tasks could be user friendly (fast pace) and have a very general use (high penetration). However, it would not make policymakers rethink their understanding of the system they are trying to intervene or the design or implementation of their policy instruments. Hence, its profundity is shallow. On the opposite side of the 3Ps space I have placed agent-level models such as ABMs and DSGEs, which push policymakers to careful think about interventions and transmission channels (deep profundity), but these are highly specialised (lower penetration) and require substantial training (slow pace). Figure 1 is an illustration about how one may think about a specific adoption. However, the placement of different methods and technologies may vary depending on the context and nature of the application as some institutions may be better able to deploy a specialised tool at a faster pace than others (e.g., because there is a long-standing tradition of training its teams in certain methods).

Figure 1. Example of analytic methods in the 3Ps space

A few clarifications about the 3Ps space are in order. First, the location of a tool or method in the 3Ps space is not static. A tool may become easier to adopt as it matures or as its adopter builds more internal capacity, just as it happened with DSGEs after years of investment in R&D by central banks, or with LLMs in recent years through private-sector investments. Hence, technologies may move across the 3Ps space. It also means that there is such thing as the right time to adopt AI or other technologies. Second, the location in the 3Ps is not normative. The choice of adopting a tool with shallow profundity is a legitimate decision if this choice is coherent with the organisation’s priorities and the problem at hand. Third, Figure 1 suggests that profundity correlates negatively with penetration and pace, i.e. a faster pace and lower penetration is likely to be accompanied by shallow profundity. While this may be true to some extent, the real world is more complex, and Figure 1 only provides a stylised picture. In reality, the colour gradient describing profundity is more nuanced or spotty. There could be tools offering both great depth of profundity while being easily accessible and of general use (causal inference tools through directed acyclic graphs is a potential candidate for a spot like this). Fourth, the degree of profundity is not determined by the complicatedness of the tool. Case in point are LLMs, which are highly complicated but mature enough to cater to a broad set of tasks through user-friendly protocols. Overall, the 3Ps space should be thought of as a synthesis to characterise a potential adoption after most of the suggested questions have been answered. It provides an aggregate overview of where candidate (perhaps competing) technologies are situated with regards to the problem at hand.

Conclusions

Hypes about emerging analytical tools and methods are not new, but the current hype for AI has unique characteristics that call for rethinking how public institutions adopt this technology for policymaking. In this adoption process, it is easy to believe promises built on inflated expectations, buying into the illusion that a particular tool will solve fundamental long-standing policy problems, and ending up with a digital elephant providing modest improvements at a high cost. Institutional contexts, incentive misalignments, short-term views, and misarranged policy priorities contribute to the numerous vicissitudes that can lead the AI adoption process to a dead end. Hence, understanding the problem, the context, and the tools are essential steps to be taken on at early stages of the adoption process. Through a hypothetical case, a set of questions to be asked, and the 3Ps space, this paper provides some pointers that may help policymakers successfully navigating the adoption of AI.

Overall, the emergence of new technologies and their potential application for policymaking is always challenging and mined with uncertainties. Nevertheless, it must be clear that adoptions are more likely to succeed when internal capacity of public entities is strengthened, prioritising long-term plans over the short-term goals that may fall prey to the illusions born in a hyped environment. Therefore, governments considering the deep integration of these kinds of tools should carefully consider the real challenges and benefits of embarking in this process before letting a hype rush them into MOUs and lavish yet incomplete contracts with service providers that may not understand societal problems from a policymaking perspective and may not have the public interest at heart.

References