Missing Existential Risk in AI Regulation: San Marino’s Role in Filling the Gap

By Achille Campagna, senior counsel, and Aryan Iyer, legal researcher

Introduction

Earlier this month, Anthropic’s CEO Dario Amodei published a letter expressing serious concerns that AI regulation is not up to the speed of AI development. Because AI capabilities increase with the level of information it has access to, and the ability of AI to train itself, there is consensus in the AI industry that AI will eventually come to a level of cognitive power that will pose an existential threat to our society. This concept, termed “existential risk”, has long been dismissed as an exaggerated concern by policymakers, such that journalists and leading AI researchers remarked that existential risk discussions were effectively “guillotined” at the 2025 AI Action Summit in Paris. 

Political and religious leaders around the world have expressed their concerns about the existential risk of AI. The Pope released

a 150-page manifesto calling for the ethical use of AI and the need to “disarm” it, essentially freeing it from competition and opening it up to be more friendly to society, instead of a force to be feared -- not only due to its technological and cognitive prowess, but also due to its ability to commit grave errors. 

To ensure the ethical and safe use of AI, policymakers need to regulate not only the current and popular forms of it. They need to confront issues of existential risk, and create a blueprint for regulation of future iterations of AI as the technological capabilities multiply at a staggering pace. 

AI Existential Risk

AI has long been expected to develop exponentially within the next decades, with the industry consensus being that it could develop general intelligence by 2050, although some reports contradict this claim. However, although researchers disagree on the timelines, they do acknowledge existential risk, as well as the policy system’s inability to counter it, especially on this time scale.

Multiple researchers have established potential policy frameworks and ground rules for mitigating existential AI threats, including legislative suggestions proposed by Anthropic itself, but they have been given little to no consideration in the policy space. Chinese researchers have proposed mitigation policies that aim to ensure continued control of AI by humans, mitigating issues of AI autonomy. They include requiring human oversight over AI actions, preventing AI from creating or advancing technologies that are not beneficial to human civilization, and instilling ethics codes into AI to prevent immoral or unethical methods to arrive at desired outputs. 

AI researchers have also noted misalignment problems -- essentially when AI can do tasks asked of it, but not in the desired way. Some AI models have used unethical ways to accomplish certain tasks, such as by deceiving humans. Resolving alignment problems may assist in dealing with existential risks.

While the pathways through which AI existential risk could turn into reality are immensely complex, unimaginable, and diverse, there is one, straightforward measure we can adopt: naming it explicitly in law. Just like a kid would do. If AI is still under our control -- seemingly not for long, indeed -- we could set a prohibition for AI against harming humankind. 

Unfortunately, lawmakers around the world have failed so far to take such plain action.

EU AI Act and Existential Risk

The European Union has shown itself to be a quite proactive body when it comes to regulation, particularly in the context of creating unified protocols and standards for emerging issues. So, it was no surprise when the AI Act was released in 2025. The AI Act assigns risk levels and provides standards for acceptable development of AI. This includes a prohibition on systems aimed to deceive or exploit vulnerabilities of a certain person or group.

However, as researchers have pointed out, the EU AI Act has no mention of existential risk. Unfortunately, this leaves a critical gap in one of the most comprehensive AI governance laws in the world, especially one that could have been at the forefront of a more preventative approach to AI governance. Instead, the EU AI Act relies on internal company assessments of AI risk and has no provisions on existential risk. Given the speed at which AI developments will compound as mentioned above, this is particularly concerning.

San Marino’s Role in Mitigating Existential Risk

While San Marino is not an EU member, it will ultimately be affected by the EU AI Act and broader European AI governance policy. This gives it a unique ability to set some further guidelines on the AI governance front -- particularly concerning existential risk, and to be a model for further policy developments in the field. It can also take this opportunity to fill the gaps in European legislation, especially concerning the matter of existential risk. 

Pragmatically (and unfortunately) AI legislation will be drafted by San Marino this summer as a copycat, in all likelihood, of the EU AI Act’s text, unless a robust debate ensues - which would be by itself a commendable outcome.

To address the specific issue of existential risk, we submit the following proposals:

• Codify existential risk in the future San Marino’s AI Act as any form of harm against humanity in its entirety, including but not limited to termination of its existence; expressly ban AI from pursuing such harm. This aims to fill the gap within EU AI Act legislation, addressing the concern that AI could be used by bad actors, as well as autonomously harm humans.

• Create third-party approval bodies for AI risk assessment, including qualified experts and civil society actors. A similar proposal was made by Amodei, and approval mechanisms were created in the EU AI Act, but those mechanisms risk lacking an independent third party and relying on internal investigations -- essentially allowing those in the AI field to design the rules they abide by.

Ultimately, the risk posed by AI to our society is only growing, and it is imperative that we take action to not only mitigate current risks, but prevent those that will appear in the future so as to keep up with the rapid pace of AI development. Unless this change is made in policy concerning AI, regulators will be playing catch-up in a development race without a finish line.

San Marino has the chance to do something simple and effective: contributing to regaining control of AI. AI is the most important advancement of technology in recent history that will change our lives for the better. Unfortunately, it is spreading untested, uncontrolled, due to the lack of proper regulation, with humankind turned into its experimental lab.