This paper summarizes a seminar presentation that I gave at the Geary Institute for Public Policy in University College Dublin, which was entitled, Failure and Learning in Policy Innovation. The talk and this paper analyzes the questions, why do governments so often fail to act on problems they can see coming? And why does meaningful policy reform tend to arrive only after things have already gone wrong? These are questions that should concern anyone interested in how public policy is made and improved. Drawing on the concept of the Innovator’s Dilemma—originally developed in the field of business management—this paper argues that failure is frequently a precondition for innovation in the public sector. Far from being simply a mark of incompetence, policy failure can serve as a powerful catalyst for the kind of learning and political mobilisation needed to bring about genuine change.
This argument has practical implications for policymakers, public managers, and citizens alike. It suggests that the policy process has built-in tendencies that delay innovation, and that understanding these tendencies is the first step towards designing institutions and processes that can overcome them. The analysis draws on concepts from three related fields—business management, policy learning, and the study of policy failure—to offer an argument that is both theoretically grounded and practically relevant.
The Innovator’s Dilemma: from business to government
The concept of the Innovator’s Dilemma was introduced by Clayton Christensen in his influential 1997 book of the same name. Christensen observed that even highly successful, well-managed companies often fail when confronted with disruptive technologies—innovations that initially appear inferior or serve only small niche markets, but eventually overtake established products. The reason, Christensen argued, is structural: companies are organised to serve their existing customers and protect their current market position. Senior managers focus on improving what already works rather than investing in unproven alternatives. By the time the disruptive technology matures, it is too late to catch up (Trein and Vagionaki 2024).
The business literature has explored various responses to this dilemma. Some scholars suggest that companies should create autonomous units to develop disruptive innovations independently, shielded from the biases of the parent organisation. Others advocate for “ambidexterity”—the capacity to exploit existing products while simultaneously exploring new ones. Strategic alliances and open innovation through crowdsourcing have also been proposed. Yet all of these solutions are difficult to implement in practice, because they require organisations to act against their own established routines and incentive structures.
The parallel with the public sector is striking. Governments, too, are complex organisations that tend to maintain established practices. Elected officials and public managers are typically rewarded for stability and continuity rather than for risky experimentation. When a new and complex policy challenge emerges—whether it is climate change, digitalisation, or a public health crisis—the incentives within the political system often favour incremental adjustments over bold innovation. The result is that governments, much like established firms, may recognise a problem but still fail to respond in time. And unlike firms, governments cannot simply be replaced by more agile competitors: they must reform from within, using the very structures and processes that created the problem in the first place (Trein and Vagionaki 2024).
Why the public sector struggles to innovate
Three interconnected factors help explain why an Innovator’s Dilemma arises in public governance. Together, they create a systematic bias towards the status quo, even when evidence points clearly towards the need for change.
Voter myopia
Research on democratic accountability shows that voters tend to evaluate their elected representatives based primarily on recent events and impressions. This short-sightedness has real consequences for policy innovation. Voters are less likely to reward politicians for taking preventive action—for instance, investing in disaster preparedness or long-term public health measures—because the benefits are diffuse and may not be felt until well after the next election. Politicians, being rational actors, respond accordingly: they focus on visible, short-term initiatives rather than the kind of forward-looking reforms that might address emerging challenges before they become crises. Recent empirical research confirms that politicians tend to hold more conservative policy preferences than their own constituents, a finding consistent with the incentives created by voter myopia (Trein and Vagionaki 2024).
Blame avoidance
A well-established body of research in political science demonstrates that a primary motivation for politicians is to avoid being blamed for policy failures. Because voters are more sensitive to losses than to gains, decision-makers have strong incentives to play it safe. Innovative policies are inherently uncertain—they may work, but they may also produce unexpected side effects or prove unpopular. In this context, the rational strategy for a politician is often to maintain existing policies, even when those policies are clearly inadequate. When problems become too visible to ignore, decision-makers may still prefer reactive measures over preventive ones, since responding to a crisis at least demonstrates responsiveness. In some cases, politicians may deny the problem altogether and delay policy responses for as long as possible (Trein and Vagionaki 2024).
Ill-structured policy problems
Many of the most pressing challenges facing modern governments are what policy scholars call “ill-structured” problems: challenges where the causes are complex, the effects are uncertain, and there is no straightforward solution that can be tested in advance. Climate change is a classic example. Everyone may agree that action is needed, but the specific measures required are fiercely debated, and their effects are difficult to predict. When a policy problem is ill-structured, it becomes even harder for decision-makers to justify the risks of innovation to a sceptical public. Strong interest groups—such as those in the tobacco or fossil fuel industries—are particularly likely to oppose preventive measures when the problem is poorly defined and its consequences are uncertain. This reinforces the tendency to stick with familiar approaches, even when they are failing (Trein and Vagionaki 2024).
These three factors—voter myopia, blame avoidance, and ill-structured problems—interact to produce a dynamic very similar to the Innovator’s Dilemma in business. Governments overvalue the status quo, underinvest in innovative responses, and often act only when the costs of inaction become impossible to ignore.
Failure as a driver of innovation
If the logic above is correct, then an uncomfortable but important conclusion follows: policy failure may often be a necessary step on the road to meaningful innovation. When existing policies visibly fail—whether because they do not solve the problem they were designed to address (programmatic failure), because the political process cannot produce agreement on a new approach (political failure), or because the policymaking process itself breaks down (processual failure)—the resulting crisis can create the political conditions needed for change.
Failure does this in several ways. It generates public attention and media scrutiny, making it harder for decision-makers to continue ignoring the problem. It provides concrete evidence that the current approach is inadequate, undermining the arguments of those who defend the status quo. And it can shift the political calculus: once a policy has visibly failed, the risks of doing nothing may outweigh the risks of trying something new. In short, failure politicises an issue and creates legitimacy for reform.
This dynamic can be seen across many policy domains. Consider the response to the 2009–10 financial crisis in the European Union. Before the crisis, EU economic governance relied heavily on soft coordination mechanisms that had limited enforcement power. The crisis exposed the inadequacy of these instruments in dramatic fashion. It was only after this failure that EU member states agreed to far-reaching institutional reforms, including stronger fiscal oversight and new bailout mechanisms. The pattern is similar in other areas: tobacco control policies were strengthened only after decades during which the health consequences of inaction became undeniable, and significant climate policy reforms tend to follow visible environmental disasters or the failure of earlier, more modest measures. Research on policy evaluation reinforces this argument: insights from evaluations often remain unused until the policy process makes them politically relevant—typically in the wake of visible failure (Trein and Vagionaki 2024).
A case in point: Switzerland’s Electronic Health Record
The development of a national Electronic Health Record (EHR) in Switzerland illustrates this dynamic in a more granular way. Beginning in 2007, the Swiss federal government established a collaborative platform called eHealth Suisse to bring together stakeholders—cantons, hospitals, professional associations, technology providers, and federal officials—to design a framework for a national EHR. During the first phase (2007–2013), the process was dominated by technical, policy-oriented learning: experts worked on defining standards and building consensus on how the system should work technically. However, learning about how to build political support for the project—what researchers call “power-oriented learning”—remained limited.
In the second phase (2013–2017), the focus shifted to the politics of passing a national EHR law. The resulting legislation, adopted in 2015, represented a political compromise rather than a technically optimal solution. General practitioners were not required to participate, the system was not particularly user-friendly, and the financing model remained unclear. Key design decisions were shaped more by the need to avoid a popular referendum than by technical considerations. When implementation began after 2018, these shortcomings became apparent: the system proved difficult to use in practice, and by 2023 only a tiny fraction of eligible individuals had opened an EHR.
It was only after this programmatic failure became visible that a new phase of reform began. Since 2021, the Swiss government has launched a comprehensive revision of the EHR framework, moving to an opt-out model, expanding participation requirements to include all ambulatory care personnel, and redesigning the system to be more functional. The failure of the initial approach created the political impetus for reforms that would have been very difficult to achieve in the first place, when the problem was still abstract and stakeholders had not yet experienced the consequences of a flawed design. The Swiss case also highlights the importance of pursuing technical and political learning in parallel rather than sequentially: when one runs ahead of the other, the resulting policy is likely to be either technically sound but politically unviable, or politically acceptable but functionally flawed (Trein, Presset, and Vagionaki 2025).
Not the only way: alternative pathways to policy innovation
It would be too fatalistic to conclude that failure is the only route to innovation in the public sector. There are other, more proactive pathways that can generate new policy ideas without waiting for things to go wrong. Policy labs, for instance, create dedicated spaces within government for experimentation and prototyping, allowing new approaches to be tested on a small scale before being rolled out more broadly. Innovation by lower levels of government—cities, regions, or cantons—can serve as a form of learning through diffusion, where successful local experiments are adopted elsewhere. Policy entrepreneurship by committed individuals within or outside government can push new ideas onto the agenda. And co-creation initiatives that bring together citizens, civil society, and public officials can generate solutions that are both innovative and grounded in practical experience.
These alternative pathways are valuable, but they face their own challenges. Innovations developed in labs or at the local level must still be scaled up and integrated into mainstream governance, which requires political support and institutional capacity. Co-creation processes must grapple with questions of legitimacy and democratic accountability. And policy entrepreneurs, however talented, still operate within the broader political environment shaped by voter myopia and blame avoidance. In many cases, these alternative pathways work best when they complement the dynamic described above—providing ready-made solutions that can be adopted when failure finally opens the political window for reform (Trein and Vagionaki 2024).
Implications for policymakers
What does all this mean in practice? Several lessons stand out.
First, policymakers should adopt a realistic rather than idealistic view of how evidence and learning feed into the policy process. The aspiration of “evidence-based policymaking” is admirable, but it does not fully reckon with the political dynamics that determine when and how evidence is used. Understanding that failure often precedes reform can help practitioners plan accordingly—for instance, by building evaluation mechanisms into new policies from the start, so that evidence of failure (or success) is available when it is needed.
Second, it is important to play the long game. Major policy reforms often require sustained effort over years or even decades. The Swiss EHR case shows that the failure of an initial approach, while painful, can lay the groundwork for a more ambitious and effective reform down the line. Practitioners and advocates who understand this dynamic are better placed to persist through setbacks and to position themselves to take advantage of the political openings that failure creates.
Third, governments should invest in the alternative pathways to innovation—policy labs, local experimentation, co-creation—not as substitutes for failure-driven reform, but as complements to it. These mechanisms can develop and test new ideas in advance, so that when a political window opens, there are credible, well-designed alternatives ready to be adopted.
Finally, the analysis underscores the importance of institutions that can bridge the gap between technical knowledge and political feasibility. Collaborative innovation processes must attend to both dimensions simultaneously. The Swiss EHR experience shows what happens when these two types of learning proceed sequentially rather than in parallel: the first attempt produced a technically limited and politically compromised design. Governance arrangements that encourage concurrent progress on technical and political fronts—what might be called parallel learning loops—offer a more promising model for public sector innovation.
Acknowledgements
This paper is based on a presentation given at the UCD Geary Institute for Public Policy as part of the Inspiring Policy Innovation seminar series. It draws on research published in Policy & Politics (Trein and Vagionaki 2024) and Government Information Quarterly (Trein, Presset, and Vagionaki 2025).
References
Trein, P., Presset, B., & Thenia Vagionaki, T. (2025). Parallel learning loops in collaborative innovation: Insights from digital government, Government Information Quarterly, Volume 42, Issue 4, 2025, https://doi.org/10.1016/j.giq.2025.102080.
Trein, J. Philipp & Vagionaki, Thenia. (2024). Why policy failure is a prerequisite for innovation in the public sector. Policy & Politics. 52(1). DOI:10.1332/03055736Y2023D000000012
About the Author
Philipp Trein is an Assistant Professor at the Institute of Political Studies at the University of Lausanne, Switzerland. His research focuses on policy learning, public sector innovation, policy coordination as well as AI in the public sector and health policy. More information can be found here: https://people.unil.ch/josefphilipptrein/