Thanks, David! Strongly upvoted.

To clarify, are those numbers relative to the people who got to know about EA in 2023 (via 80,000 Hours or any source)?

Thanks for following up, Matthew.

Plenty of people want wealth and power, which are "conducive to gaining control over [parts of] humanity".

I agree, but I think very few people want to acquire e.g. 10 T$ of resources without broad consent of others. In addition, if a single AI system expressed such a desire, humans would not want to scale up its capabilities.

I agree with Robin Hanson on this question. However, I think humans will likely become an increasingly small fraction of the world over time, as AIs become a larger part of it. Just as hunter-gatherers are threatened by industrial societies, so too may biological humans one day become threatened by future AIs. Such a situation may not be very morally bad (or deserving the title "existential risk"), because humans are not the only morally important beings in the world. Yet, it is still true that AI carries a great risk to humanity.

I agree biological humans will likely become an increasingly small fraction of the world, but it does not follow that AI carries a great risk to humas^[1]. I would not say people born after 1960 carry a great risk risk to people born before 1960, even though the fraction of the global resources controlled by the latter is becoming increasingly small. I would consider that AI poses a great risk to humans if these were expected to suffer significantly more than in their typical lives, which also involve suffering, in the process of losing control over resources.

1. ^{^}

   You said "risk to humanity" instead of "risk to humans". I prefer this because humanity is sometimes used to include other beings.

Hi David,

There is a significant overlap between EA and AI safety, and it is often unclear whether people supposedly working on AI safety are increasing/decreasing AI risk. So I think it would be helpful if you could point to some (recent) data on how many people are being introduced to global health and development, and animal welfare via 80,000 Hours.

Thanks for the clarifications, Joel.

At the end of the day, I apologize if the publication was not in line with what EA Funds would have wanted - I agree it's probably a difference in norms. In a professional context, I'm generally comfortable with people relaying that I said X in private, unless there was an explicit request not to share

It is clear from Linch's comment that he would have liked to see a draft of the report before it was published. Did you underestimate the interest of EA Funds in reviewing the report before its publication, or did you think their interest in reviewing the report was not too relevant? I hope the former.

Thanks for the comment, Richard.

So I don't think the headline numbers are very useful here (especially because we could make them far far higher by counting future lives).

I used to prefer focussing on tail risk, but I now think expected deaths are a better metric.

• Interventions in the effective altruism community are usually assessed under 2 different frameworks, existential risk mitigation, and nearterm welfare improvement. It looks like 2 distinct frameworks are needed given the difficulty of comparing nearterm and longterm effects. However, I do not think this is quite the right comparison under a longtermist perspective, where most of the expected value of one’s actions results from influencing the longterm future, and the indirect longterm effects of saving lives outside catastrophes cannot be neglected.
• In this case, I believe it is better to use a single framework for assessing interventions saving human lives in catastrophes and normal times. One way of doing this, which I consider in this post, is supposing the benefits of saving one life are a function of the population size.
• Assuming the benefits of saving a life are proportional to the ratio between the initial and final population, and that the cost to save a life does not depend on this ratio, it looks like saving lives in normal times is better to improve the longterm future than doing so in catastrophes.

Thanks for pointing that out, Ted!

1/10,000 * 8 billion people = 800,000 current lives lost in expectation

The expected death toll would be much greater than 800 k assuming a typical tail distribution. This is the expected death toll linked solely to the maximum severity, but lower levels of severity would add to it. Assuming deaths follow a Pareto distribution with a tail index of 1.60, which characterises war deaths, the minimum deaths would be 25.3 M (= 8*10^9*(10^-4)^(1/1.60)). Consequently, the expected death toll would be 67.6 M (= 1.60/(1.60 - 1)*25.3*10^6), i.e. 1.11 (= 67.6/61) times the number of deaths in 2023, or 111 (= 67.6/0.608) times the number of malaria deaths in 2022. I certainly agree undergoing this risk would be wild.

Side note. I think the tail distribution will eventually decay faster than that of a Pareto distribution, but this makes my point stronger. In this case, the product between the deaths and their probability density would be lower for higher levels of severity, which means the expected deaths linked to such levels would represent a smaller fraction of the overall expected death toll.

Thanks for elaborating, Joseph!

A major consideration here is the use of AI to mitigate other x-risks. Some of Toby Ord's x-risk estimates

I think Toby's existential risk estimates are many orders of magnitude higher than warranted. I estimated an annual extinction risk of 5.93*10^-12 for nuclear wars, 2.20*10^-14 for asteroids and comets, 3.38*10^-14 for supervolcanoes, a prior of 6.36*10^-14 for wars, and a prior of 4.35*10^-15 for terrorist attacks. These values are already super low, but I believe existential risk would still be orders of magnitude lower. I think there would only be a 0.0513 % (= e^(-10^9/(132*10^6))) chance of a repetition of the last mass extinction 66 M years ago, the Cretaceous–Paleogene extinction event, to be existential. I got my estimate assuming:

• An exponential distribution with a mean of 132 M years (= 66*10^6*2) represents the time between i) human extinction in such catastrophe and ii) the evolution of an intelligent sentient species after such a catastrophe. I supposed this on the basis that:
  • An exponential distribution with a mean of 66 M years describes the time between:
    • 2 consecutive such catastrophes.
    • i) and ii) if there are no such catastrophes.
  • Given the above, i) and ii) are equally likely. So the probability of an intelligent sentient species evolving after human extinction in such a catastrophe is 50 % (= 1/2).
  • Consequently, one should expect the time between i) and ii) to be 2 times (= 1/0.50) as long as that if there were no such catastrophes.
• An intelligent sentient species has 1 billion years to evolve before the Earth becomes habitable.

Thanks for sharing, Ávila! You may be interested in the post Rejection thread: stories and tips.

Do you think Open Philanthropy's animal welfare grants should have write-ups whose main text is longer than 1 paragraph? I think it would be great if you shared the cost-effectiveness analyses you seem to be doing. In your recent appearance on The 80,000 Hours Podcast (which I liked!), you said (emphasis mine):

Lewis Bollard: Our goal is to help as many animals as we can, as much as we can — and the challenge is working out how to do that.

[...]

If there’s not a track record, if this is maybe more speculative or a longer-term play, we try to vet the path to impact. So we try to look at what are the steps that would be required to get to the long-term goal. How realistic are those steps? Do they logically lead to one another? And what evidence is there about whether we’re on that path, about whether the group has achieved those initial steps? But then there is also some degree of needing to look at plans and just assess plausibly how strong do these plans look? It’s not always possible to pin down the exact numbers. We try as hard as we can to do that, though.

To be clear, the main text of the write-ups of Open Philanthropy’s large grants is 1 paragraph across all areas, not just the ones related to animal welfare. However, I wonder whether there would be freedom for a given area to share more information (in grant write-ups or reports) if the people leading it thought that to be valuable.

Do you think the nascent field of economics of animal welfare will play a significant role in increasing the spending on animal welfare interventions? To illustrate what I am referring to, I share below the abstracts of 2 pieces of research in that field.

Kevin Kuruc's paper Monetizing the externalities of animal agriculture: insights from an inclusive welfare function (2023):

Animal agriculture encompasses global markets with large externalities from animal welfare and greenhouse gas emissions. We formally study these social costs by embedding an animal inclusive social welfare function into a climate-economy model that includes an agricultural sector. The total external costs are found to be large under the baseline parameterization. These results are driven by animal welfare costs, which themselves are due to an assumption that animal lives are worse than nonexistence. Though untestable—and perhaps controversial—we find support for this qualitative assumption and demonstrate that our results are robust to a wide range of its quantitative interpretations. Surprisingly, the environmental costs play a comparatively small role, even in sensitivity analyses that depart substantially from our baseline case. For the model to find that beef, a climate-intensive product, has a larger total externality than poultry, an animal-intensive product, we must simultaneously reduce the animal welfare externality to 1% of its baseline level and increase climate damages roughly 35-fold. Correspondingly, the model implies both that the animal agriculture sector is much larger than its optimal level and that considerations across products ought to be dominated by animal welfare, rather than climate, effects.

Romain Espinosa's Animals and social welfare (2023):

I propose a framework to evaluate the social gains from policies regarding animals. The model considers both the welfare of animals and humans. The gains in animal welfare are estimated by considering the violations of the animals’ fundamental freedoms weighted for each species. I apply this framework to twenty policy proposals targeting wild, domestic, farmed, and laboratory animals. Although the policies benefit from widespread popular support in France (the annual willingnesses-to-pay range between 15 and 39 Euros per person per year), I show that they have very heterogeneous impacts on animal welfare (valued at between 0.013 and 3618 Euros per person per year). I further show that humans’ willingness-to-pay for policies improving animal welfare is a poor predictor of the effective impact on animal welfare of these policies. I conclude that it is essential to value animal welfare per se in cost-benefit analyses in order to determine the set of welfare-increasing policies.