In recent years, Artificial Intelligence (AI) has surged to the forefront of public consciousness as a potentially transformative technology. Alongside its benefits, AI has ignited intense debate about the nature, likelihood, and severity of its [[What are the global risks and opportunities posed by advancing AI?|associated risks]]. The discourse is characterized by deep-seated disagreements among experts, policymakers, and the public, spanning from optimistic visions of AI-driven progress to dire warnings of existential threats. This post delves into the key disagreements surrounding AI risk, drawing insights from two studies by the [Forecasting Research Institute](https://forecastingresearch.org/). It explores the stark divergence in beliefs about AI's potential for catastrophic and existential risks, examining factors such as differing expectations about AI progress, economic impacts, and fundamental worldview differences. Understanding these areas of disagreement as well as the factors that contribute to them is crucial for fostering more productive discussions about AI development and safety. Ultimately, these discussions will shape how we approach the advancement and governance of this transformative technology. ### Why does this matter? The divergent perspectives on AI risk pose a significant challenge to our ability to address potential issues proactively while harnessing the technology's benefits. For example: - **Collective action**: The absence of a unified view on AI risk exacerbates collective action problems in AI development and governance. A shared understanding of the risks could align incentives and foster coordinated efforts. - **Policy and regulation**: Disagreement on AI risks hinders the creation of effective policies and regulations, both domestically and internationally, leaving a regulatory vacuum in a rapidly evolving field. - **Preparedness**: Conflicting assessments of the probability and severity of AI risks impede adequate preparation, potentially leaving society vulnerable to risks that do manifest. - **Social cohesion**: Much like the divisions witnessed during the COVID-19 pandemic, widespread disagreement about AI risk threatens to deepen societal rifts and fuel polarization. ### What are the main areas of disagreement about AI risk? %% ### #todo - [ ] Review charts below and identify the most interesting areas of relative agreement vs. disagreement. - [ ] Check whether there was more data contrasting AI-concerned vs. AI skeptics groups. %% To address this question, I'll draw upon findings from the [Existential-Risk Persuasion Tournament](https://forecastingresearch.org/news/results-from-the-2022-existential-risk-persuasion-tournament) (XPT), conducted by the [Forecasting Research Institute](https://forecastingresearch.org/) (FRI) from June to October 2022. This tournament brought together diverse groups to make probabilistic predictions about various catastrophic and existential risks, including those posed by AI. >[!The study] >**Participant groups**: The XPT gathered forecasts from three distinct cohorts: >1. **Experts**: 80 individuals with strong academic or practical backgrounds in relevant domains (e.g., AI, biorisk). >2. **[Superforecasters](https://en.wikipedia.org/wiki/Superforecaster)**: 89 individuals known for their exceptional accuracy in short-term geopolitical predictions. >3. **Public**: A reference class of college-educated individuals. > >**Tournament stages**: The tournament unfolded in four stages: >1. **Initial forecasts**: Participants provided individual predictions and explanations without knowledge of others' responses. >2. **Intra-group collaboration**: Participants worked in teams within their cohort (e.g., experts with experts) to discuss and refine forecasts. >3. **Inter-group collaboration**: Mixed teams of experts and superforecasters debated issues and updated predictions. >4. **Cross-team review**: Each team reviewed and potentially revised their forecasts after examining other teams' predictions. > >The full report is available [here](https://static1.squarespace.com/static/635693acf15a3e2a14a56a4a/t/64f0a7838ccbf43b6b5ee40c/1693493128111/XPT.pdf). > The XPT focused on two **key questions related to AI ris**k: 1. What is the chance that AI will cause human extinction (defined as a drop in the global population below 5,000 people) by [2030, 2050, 2100]? 2. What is the chance that AI will cause a global catastrophic event (defined as an event that causes the death of 10% of humanity within a five-year period) by [2030, 2050, 2100]? These questions stand out for their explicit and precise nature. Their unambiguous formulation allows for direct comparison of responses, contrasting sharply with much of the public discourse about AI risk where people often use vague and implicit notions of risk. The most striking outcome of the XPT with respect to this note was the **stark divergence in beliefs about AI risk** among participants. Let's delve into some specific findings to illustrate this disparity. #### Forecasts of AI risk The table below shows the final median forecasts of AI catastrophic and extinction risk for the three time horizons. ![[Pasted image 20240913103351.png]] A notable pattern emerges: **domain experts consistently assigned higher probabilities** to both catastrophic and extinction risks compared to superforecasters, across all timeframes. For instance, considering the likelihood of AI-induced catastrophe by 2100, the median domain expert estimate (12%) was nearly six times higher than that of the median superforecaster (2.13%). The gap widens further for extinction risk, with experts' median estimate (3%) approximately an order of magnitude greater than superforecasters' (0.38%). The **distribution of forecasts** further illuminates the spectrum of beliefs. The plots below show the forecasts from the last round of the tournament on AI catastrophic and extinction risk by 2100. We can detect a large variation in beliefs among different groups, although we also see some overlap between the groups. ![[Pasted image 20240914084548.png]] ![[Pasted image 20240914085950.png]] While there's considerable variation within each group, a clear separation exists between the most concerned and the most skeptical forecasters. To illustrate this, the XPT evaluated the results for two sub-groups: the "**AI-concerned**" and "**AI-skeptic**" participants, representing the top and bottom thirds of extinction risk forecasts, respectively. The median AI-concerned forecaster assigned a 7.5% probability to AI-driven extinction by 2100, in sharp contrast to the median AI-skeptic's 0.01% – a 750-fold difference. #### Forecasts of AI progress and economic impact The XPT extended its inquiry beyond direct AI risk, exploring forecasts on related but distinct issues: the pace of AI progress, advancements in key inputs to AI development, and the economic impact of AI. These additional forecasts provide valuable context for understanding disagreements between participants. ##### AI progress Regarding AI progress, a notable convergence emerges in the expectations of both domain experts and superforecasters. Both groups anticipate rapid advancements in AI capabilities over the coming decades, with a general consensus that "advanced AI" might emerge around mid-century and "AGI" by 2100. This **alignment in projections of AI development timelines** is particularly intriguing given the stark divergence in their assessments of associated risks discussed in the previous section. ![[Pasted image 20240914093320.png]] ![[Pasted image 20240914093330.png]] ![[Pasted image 20240914094215.png]] ##### Inputs to AI development In terms of inputs to AI development, **both groups forecast substantial increases in investment**. Interestingly, recent developments in the field have already outpaced some near-term predictions. The training of models like GPT-4 and Gemini 1.0 Ultra has [exceeded](https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models) the compute cost forecasts for 2024, underscoring the rapid and sometimes unpredictable nature of progress in this domain. ![[Pasted image 20240914094818.png]] ##### Economic impacts Perhaps one of the most striking disparities emerges in the forecasts of AI's economic impact. While both groups generally agree on the trajectory of AI capabilities, their **expectations of economic consequences differ dramatically**. The median superforecaster assigned a mere 2.75% probability to annual GDP growth exceeding 15% before 2100. In stark contrast, the median AI domain expert estimated a 25% chance of such unprecedented economic expansion—a nearly tenfold difference. ![[Pasted image 20240914095421.png]] %% #### So who's right? Since this isn't the main focus of this note, and due to time constraints, I will largely have to skip this question. However, a few quick remarks: - There is very little research on whether people are accurate at forecasting low-probability events.^[See [this interview](https://80000hours.org/podcast/episodes/ezra-karger-forecasting-existential-risks/)with Ezra Karger, the lead author on the XPT report.] Because such events occur very rarely by definition, it's very hard to test whether people are accurate at forecasting them. - Superforecasters are accurate at forecasting geopolitical questions on short time horizons. On the other hand, experts are people at the frontier of changes, so they might see things that others don't. If you expect the world to change fast over the coming decades, you might give more weight to the forecasts by experts. Otherwise, you might want to defer more to superforecasters. - P14: Superforecasters may rely more on base rates (data on similar cases) than domain experts, which might be misleading or unavailable in the case of long-run risks. - What's the relevant knowledge to assess such risks, and who possesses this knowledge? - What's the appropriate style of thinking and reasoning? - P20: "How should we interpret this persistent divide between groups? One possibility is that the experts are biased toward the topics they are professionally invested in and overweight the tail-risks they spend time thinking about—perhaps partly because those most worried about existential risk opt to dedicate their lives to studying it. Another is that the superforecasters are skilled at using historical data for relatively short-run forecasts but might struggle to adapt their methods to longer-run topics with less data—even when they have experts on hand to walk them through the topic. It is also possible that the epistemic strategies that were successful in earlier short-run forecasting tournaments, when the superforecasters attained their status, are not as appropriate at other points in time. For example, base rates may be more useful in periods of relative geopolitical calm and less useful in periods of greater conflict. All of these possibilities may be operating together, to various degrees." %% ### What are the roots of those disagreements? To explore the factors contributing to disagreements about AI risk, we'll examine another FRI study: "[Roots of Disagreement on AI Risk](https://forecastingresearch.org/ai-adversarial-collaboration)". This research builds upon the XPT findings, delving deeper into the beliefs of forecasters with varying levels of concern about AI risk. >[!The study] >**Participant selection**: >- 11 AI-skeptical individuals from the XPT >- 11 domain experts concerned about AI risk > >**Study process**: >- Over the course of 8 weeks, members of both groups read background materials, developed forecasts, and engaged in online discussions. >- The median AI skeptic spent 80 hours on this process, the median AI-concerned participant spent 31 hours. > >**Study objectives**: >- Assess potential convergence of beliefs through extensive engagement with opposing arguments. >- Identify near-term indicators (resolving by 2030) as well as long-term indicators that explain disagreements between groups. >- Evaluate how risk forecasts would change based on the resolution of short-term questions. > >The full report is available [here](https://static1.squarespace.com/static/635693acf15a3e2a14a56a4a/t/65ef1ee52e64b52f145ebb49/1710169832137/AIcollaboration.pdf). The researchers proposed four hypotheses to explain the stark divergence in views on AI risk between the two groups. Let's examine each of these hypotheses in turn, exploring how they might illuminate the roots of disagreement in AI risk assessment. #### Hypothesis 1: Lack of engagement with differing viewpoints The first hypothesis considered whether the divergence in AI risk assessments could simply stem from a lack of exposure to diverse viewpoints. This hypothesis suggests that increased engagement with opposing arguments and broader access to information might bridge the gap between AI skeptics and those concerned about AI risks. However, the results painted a different picture. The median AI skeptic's forecast of existential catastrophe due to AI by 2100 barely shifted, moving from 0.10% at the start to 0.12% at the conclusion of the project. This minute change suggests that exposure to arguments from AI-concerned experts did little to alter the skeptics' fundamental risk assessment. Conversely, the median AI-concerned expert's forecast saw a more noticeable change, decreasing from 25% to 20%. Yet, intriguingly, they attributed this adjustment not to the inter-group discussions but to external factors—namely, increased attention to AI risk from policymakers and the public during the study period. The **negligible convergence between these groups**, despite intensive engagement, challenges the initial hypothesis. It suggests that the roots of disagreement on AI risk run deeper than mere information asymmetry or lack of exposure to opposing viewpoints. #### Hypothesis 2: Different short-term expectations about AI The study explored a second hypothesis to explain the persistent divergence in AI risk assessments: Could these disagreements stem from differing expectations about AI developments in the near future? This proposition suggests that as short-term events unfold and uncertainties resolve, we might see a convergence of long-term risk assessments between AI skeptics and those concerned about AI risks. To test this hypothesis, participants were asked to anticipate how their long-term AI risk forecasts might change based on the resolution of various short-term questions, most of which would be answered by 2030. This method aimed to gauge the potential for future convergence as concrete near-term outcomes materialize.^[It's worth noting that this methodology relies on participants' ability to accurately predict how they would update their views given new information—a skill that itself is subject to debate.] However, the study mostly found evidence against this hypothesis. Participants generally indicated that they expected minimal updates to their 2100 AI risk forecasts based on how these short-term indicators might resolve. Even the most influential short-term indicator—whether [METR](https://metr.org/) or a similar group would identify dangerous AI capabilities by 2030—was expected to narrow the substantial initial gap between the two groups by a mere 1.2 percentage points. This finding is particularly striking given the approximately 25 percentage point initial divergence between the groups' risk assessments. It suggests that the **short-term indicators examined in this study explain only a fraction of the disagreement about long-term AI risk**. Instead, the results point to more fundamental differences in how these groups approach risk assessment or in their underlying assumptions about AI development and its potential consequences. #### Hypothesis 3: Different long-term expectations about AI Could divergent expectations about AI developments over the coming decades and centuries explain the persistent disagreements about AI risk? The findings offer a nuanced perspective on this question, revealing both areas of convergence and enduring points of contention. Interestingly, the research uncovered a trend towards **convergence when considering longer time horizons**. For instance, when assessing the likelihood of an AI-caused catastrophic event over the next millennium, the gap between skeptics and concerned participants narrowed significantly. The median skeptic assigned a 30% probability, while the median concerned participant estimated 40% – a much smaller disparity (in ratio terms) compared to their near-term risk assessments. However, **substantial disagreements persisted**, particularly regarding AI risk by 2100. The most stark divergences centered on the likelihood of AI intentionally or unintentionally causing human extinction (12% and 3% for the median concerned participant, respectively, and 0.02% and 0.01% for the median skeptic, respectively), as well as the potential for human misuse of AI leading to extinction (0.5% for the median concerned, 0.03% for the median skeptic). In each case, concerned participants assigned probabilities orders of magnitude higher than their skeptical counterparts. Another notable point of disagreement emerged regarding the timeline for AI dominance. When asked to predict the year by which AI would displace humans as the primary force shaping future events, the median estimates differed by over four centuries – 2450 for skeptics versus 2045 for concerned participants. The study identified four key **drivers of these long-term disagreements**: - **Timelines**: Skeptics emphasized the need for fundamental breakthroughs and advanced robotics for AI to pose an existential threat and expected a delay in adoption and deployment of powerful AI systems. Meanwhile, concerned participants highlighted recent rapid progress and the potential for AI to accelerate its own development, and a belief that robotics won't be necessary for AI to have a large impact on the physical world. - **Goals that incentivize killing everyone**: Skeptics viewed dangerous goals as a small subset of possible AI objectives. By contrast, concerned participants worried about instrumental convergence and goal misgeneralization, and expected that most goals an AI might have would likely benefit from human extinction to access and avoid competition for resources. - **Difficulty of killing everyone**: Skeptics doubted the feasibility of global human extinction, while concerned participants identified several plausible mechanisms for large-scale devastation, including novel pathogens, nano-bots, or nuclear catastrophes. - **Societal response**: Skeptics were more optimistic about society's ability to recognize and mitigate AI risks. Concerned participants worried about short AI timelines and competitive pressures limiting safety measures, and emphasized that containment of dangerous AIs would need to work 100% of the time, since a single failure of containment might be enough to cause catastrophe. These findings suggest that while there's some convergence over very long time horizons, significant disagreements persist about the nature, likelihood, and timeline of potential AI-driven catastrophes. #### Hypothesis 4: Fundamental worldview disagreements The study into AI risk perceptions unearthed a compelling insight: the root of disagreements often lies not in the specifics of AI technology, but in fundamental differences in worldviews. These disparities profoundly shape how different groups perceive and evaluate the potential risks associated with advanced AI. At the core of these divergences are contrasting **approaches to reasoning and forecasting**. Proponents of AI risk tend to favor deductive reasoning,^[Deductive reasoning is a top-down approach: It starts with general principles, theories, or premises and moves toward a specific conclusion that follows if the premises are true.] constructing theoretical models to predict specific catastrophic outcomes. They often adopt an "inside view,"^[The inside view focuses on the specific details and unique aspects of the situation at hand. It builds a model or narrative about how events might unfold based on all available information about the particular case. The inside view updates its forecasts more quickly based on new information about the situation at hand.] focusing on the unique aspects of AI as an unprecedented technology. In contrast, skeptics lean towards inductive reasoning,^[Inductive reasoning is a bottom-up approach: It starts with specific observations and moves toward generalizations and conclusions.] drawing on historical data and taking an "outside view"^[The outside view looks at a situation from a broader perspective, seeing it as an instance of a broader reference class of similar cases. It abstracts away from the specific details of the situation at hand and focuses on the "base rates" in analogous situations based on historical data. The outside view provides a stable benchmark and updates its forecasts more slowly based on new case-specific information.] that places AI within a broader context of technological advancements.^[My account of how each side tends to align more with one view than another maybe be right on average, but there are certainly exceptions. For example, proponents may take the outside view on how the arrival of a new and more intelligent species (i.e., Homo sapiens) leads to the extinction of less intelligent species, and that we should thus be worried about creating AGI and ASI. The disagreement in that case is over the correct reference class for the outside view ("reference class tennis"), where skeptics might argue that "AI is just another technology" and proponents might argue that "AI is like a new invasive species that's more fit than the incumbent."] These epistemological preferences are further complemented by differing **attitudes towards complexity, uncertainty, and change**. Proponents generally believe in our capacity to reason about long-term events in a complex world and anticipate rapid, discontinuous change and high variance in outcomes, such that both very good and very bad outcomes for humanity seem plausible. Skeptics, however, emphasize the unpredictability of long-term outcomes and expect more moderate, continuous change with less variance, such that things largely continue as they have. The groups also diverge in their **attitudes towards risk management**. Proponents advocate for caution in the face of uncertainty, aligning with the [precautionary principle](https://en.wikipedia.org/wiki/Precautionary_principle). Skeptics, on the other hand, focus more on potential benefits and worry about stifling innovation, following the [proactionary principle](https://en.wikipedia.org/wiki/Proactionary_principle). The two sides fundamentally disagree about whether the burden of proof falls on those who want to develop AI systems (demonstrating that AI is safe) or those who want to impose restrictions (demonstrating that AI is unsafe). Moreover, there are stark contrasts in **trust towards technological progress and societal structures**. While proponents highlight the potential for devastating effects from certain new technologies, skeptics view technology as a primary driver of human progress with a net positive historical impact. Similarly, proponents tend to be pessimistic about the effectiveness of existing institutions in governing advanced AI and our ability to muster a coordinated response to novel risks, while skeptics are more optimistic about societal adaptation to new technologies. Proponents emphasize the fragility of human existence, while skeptics emphasize human agency and resilience. These fundamental differences in worldviews lead to persistent disagreements because **each group evaluates the same information through their distinct lens**. Arguments from one side often fail to address the underlying principles guiding the other group's perspectives, resulting in a dialogue where the two groups frequently talk past each other. Understanding these deep-seated differences is crucial for fostering more productive discussions about AI risk. It highlights the **need for approaches that can bridge these fundamental divides**, encouraging a more holistic and nuanced dialogue about the future of AI and its potential impacts on humanity. %% **Reasoning methods: deductive vs. inductive reasoning** - **Deductive reasoning** is a top-down approach: It starts with general principles, theories, or premises and moves toward a specific conclusion that follows if the premises are true. - **Inductive reasoning** is a bottom-up approach: It starts with specific observations and moves toward generalizations and conclusions. - **Proponents of AI risk tend to apply more deductive reasoning**.^[I believe that this relationship is not absolute, and that both groups use both reasoning styles. However, I believe that the general pattern is true as discussed here.] They rely on theoretical models and abstract reasoning to predict specific catastrophic outcomes. For example, they argue that AI systems will converge towards certain instrumental goals such as power-seeking in order to achieve their terminal goals, and that this will be dangerous in sufficiently advanced AI systems. - **Skeptics of AI risk tend to apply more inductive reasoning**. They examine historical data on technological advancements and their impacts on society. Based on this data, they argue that there is no evidence of technologies causing societal-scale harm, and that we should therefore expect such outcomes to be unlikely in the future. - Conversely, proponents of AI risk tend to be dismissive of inductive reasoning in the context of AI risk. They argue that AI is unprecedented, that historical data is limited in its usefulness to predict AI outcomes, and that concrete evidence is not needed to assign a serious probability of AI risk. Furthermore, they believe that waiting for historical data to accumulate is not a good strategy, because once there is evidence of an existential threat from AI, we might all be dead. - Skeptics, on the other hand, tend to be unwilling to update much on the basis of deductive reasoning. They don't trust complex theoretical arguments to guide their reasoning. **Forecasting perspectives: inside vs. outside view** - The **inside view** focuses on the specific details and unique aspects of the situation at hand. It builds a model or narrative about how events might unfold based on all available information about the particular case. The inside view updates its forecasts more quickly based on new information about the situation at hand. - The **outside view** looks at a situation from a broader perspective, seeing it as an instance of a broader reference class of similar cases. It abstracts away from the specific details of the situation at hand and focuses on the "base rates" in analogous situations based on historical data. The outside view provides a stable benchmark and updates its forecasts more slowly based on new case-specific information. - **Proponents of AI risk tend to take the inside view**. They argue that AI is an unprecedented technology where there is no reasonable reference class. Furthermore, they caution that taking the outside view predictably leads to an underestimation of novel risks in scenarios with discontinuous change. At the same time, proponents give more weight to expert knowledge that they deem relevant to the situation. - Proponents tend to use **deductive reasoning within the inside view** to construct scenarios of how AI could pose existential threats. - **Skeptics of AI risk tend to take the outside view**. They aren't convinced by the arguments for AI as a special case that warrants a focus on the inside view, and insist that we should start with the outside view to anchor our judgment. Humanity has a long track record of survival, and new technologies have never posed existential threats. They argue that the inside view is overconfident in its models that don't account for real-world complexities. - Skeptics tend to use **inductive reasoning within the outside view** to generalize from historical patterns, leading them to the conclusion that risks are manageable or unlikely. - My account of how each side tends to align more with one view than another maybe be right on average, but there are certainly exceptions. For example, proponents may take the outside view on how the arrival of a new and more intelligent species (i.e., homo sapiens) leads to the extinction of less intelligent species, and that we should thus be worried about creating AGI and ASI. The disagreement in that case is over the correct reference class for the outside view ("**reference class tennis**"), where skeptics might argue that "AI is just another technology" and proponents might argue that "AI is like a new invasive species that's more fit than the incumbent." **Attitudes towards complexity, uncertainty, and change** - **Proponents** tend to believe in our ability to reason about and make forecasts about long-term events in a complex world. They expect the world to change relatively quickly and in a discontinuous way. Lastly, they expect more variance in outcomes and assign a higher probability to both very good and very bad outcomes from the perspective of humanity. - **Skeptics** emphasize that the world is complex and to a large extent unpredictable in the long-term. They expect the world to change more moderately and continuously. And they generally expect less variance, such that the range of plausible outcomes is more narrow and things largely continue as they have. **Attitudes towards risk** - **Proponents** tend to advocate for caution in the face of uncertainty and lack of consensus about AI risk. Following the **[precautionary principle](https://en.wikipedia.org/wiki/Precautionary_principle)**, they argue that the burden of proof that AI does not pose an existential threat to humanity falls on those developing AI systems. - **Skeptics** tend to be more focused on the potential benefits of AI and worried about stifling innovation and growth. Following the **[proactionary principle](https://en.wikipedia.org/wiki/Proactionary_principle)**, they advocate for careful risk-benefit analysis, shifting the burden of proof on those who want to place restrictions on AI to prove that it's harmful rather than requiring innovators to prove that it's safe. **Trust in technological progress** - **Proponents** tend to believe that certain new technologies might have [devastating effects on humanity](https://nickbostrom.com/papers/vulnerable.pdf), emphasizing that technological progress can be high-variance. - **Skeptics** tend to view technology as the primary driver of human progress, emphasizing that the historical impact of technology is strongly net positive for humanity. **Trust in societal structures and adaptation** - **Proponents** tend to be pessimistic about the effectiveness of existing institutions and governance structures at controlling and governing advanced AI. Additionally, they tend to be pessimistic about our ability to muster a coordinated response to novel risks, and worried that AI risks will outpace societal adaptation. In general, this perspective emphasizes the **fragility** of human existence. - **Skeptics** tend to be more hopeful about the ability of existing systems to control and regulate technological advancements, and our ability to recognize and address novel risks in the future. This perspective emphasizes human **agency and resilience**. In summary, disagreements about AI risk between the two groups arise from contrasting worldviews regarding epistemological preferences, risk tolerance, and trust in technology and human governance. These fundamental differences lead to persistent disagreements because each group evaluates the same information through their distinct lens. Both sides enter the conversation with very different foundational beliefs and reasoning styles. Arguments from one side often fail to address the underlying principles guiding the other group's perspectives, leaving the two groups talking past each other. %% --- Topics: - [[Epistemology]] - [[Artificial Intelligence]] - [[Futurism]] Related notes: - [[The AI Revolution and the Tapestry of Tomorrow (Index)]] - [[How much can I trust any of my views on AI?]] - [[Most Important Century Series by Holden Karnofsky]]