Quick takes

I spent way too much time organizing my thoughts on AI loss-of-control ("x-risk") debates without any feedback today, so I'm publishing perhaps one of my favorite snippets/threads: A lot of debates seem to boil down to under-acknowledged and poorly-framed disagreements about questions like “who bears the burden of proof.” For example, some skeptics say “extraordinary claims require extraordinary evidence” when dismissing claims that the risk is merely “above 1%”, whereas safetyists argue that having >99% confidence that things won’t go wrong is the “extraordinary claim that requires extraordinary evidence.”  I think that talking about “burdens” might be unproductive. Instead, it may be better to frame the question more like “what should we assume by default, in the absence of definitive ‘evidence’ or arguments, and why?” “Burden” language is super fuzzy (and seems a bit morally charged), whereas this framing at least forces people to acknowledge that some default assumptions are being made and consider why.  To address that framing, I think it’s better to ask/answer questions like “What reference class does ‘building AGI’ belong to, and what are the base rates of danger for that reference class?” This framing at least pushes people to make explicit claims about what reference class building AGI belongs to, which should make it clearer that it doesn’t belong in your “all technologies ever” reference class.  In my view, the "default" estimate should not be “roughly zero until proven otherwise,” especially given that there isn’t consensus among experts and the overarching narrative of “intelligence proved really powerful in humans, misalignment even among humans is quite common (and is already often observed in existing models), and we often don’t get technologies right on the first few tries.”

Not that we can do much about it, but I find the idea of Trump being president in a time that we're getting closer and closer to AGI pretty terrifying. A second Trump term is going to have a lot more craziness and far fewer checks on his power, and I expect it would have significant effects on the global trajectory of AI.

I am concerned about the H5N1 situation in dairy cows and have written and overview document to which I occasionally add new learnings (new to me or new to world). I also set up a WhatsApp community that anyone is welcome to join for discussion & sharing news. In brief: * I believe there are quite a few (~50-250) humans infected recently, but no sustained human-to-human transmission * I estimate the Infection Fatality Rate substantially lower than the ALERT team (theirs is 63% that CFR >= 10%), something like 80%CI = 0.1 - 5.0 * The government's response is astoundingly bad - I find it insane that raw milk is still being sold, with a high likelihood that some of it contains infectious H5N1 * There are still quite a few genetic barriers to sustained human-to-human transmission * This might be a good time to push specific pandemic preparedness policies

As someone predisposed to like modeling, the key takeaway I got from Justin Sandefur's Asterisk essay PEPFAR and the Costs of Cost-Benefit Analysis was this corrective reminder – emphasis mine, focusing on what changed my mind: More detail: Tangentially, I suspect this sort of attitude (Iraq invasion notwithstanding) would naturally arise out of a definite optimism mindset (that essay by Dan Wang is incidentally a great read; his follow-up is more comprehensive and clearly argued, but I prefer the original for inspiration). It seems to me that Justin has this mindset as well, cf. his analogy to climate change in comparing economists' carbon taxes and cap-and-trade schemes vs progressive activists pushing for green tech investment to bend the cost curve. He concludes:  Aside from his climate change example above, I'd be curious to know what other domains economists are making analytical mistakes in w.r.t. cost-benefit modeling, since I'm probably predisposed to making the same kinds of mistakes. 

This December is the last month unlimited Manifold Markets currency redemptions for donations are assured: https://manifoldmarkets.notion.site/The-New-Deal-for-Manifold-s-Charity-Program-1527421b89224370a30dc1c7820c23ec Highly recommend redeeming donations this month since there are orders of magnitude more currency outstanding than can be donated in future months

Metaculus launches round 2 of the Chinese AI Chips Tournament Help bring clarity to key questions in AI governance and support research by the Institute for AI Policy and Strategy (IAPS). Start forecasting on new questions tackling broader themes of Chinese AI capability like:  Will we see a frontier Chinese AI model before 2027? Will a Chinese firm order a large number of domestic AI chips? Will a Chinese firm order a large number of US or US-allied AI chips?

For a long time I found this surprisingly nonintuitive, so I made a spreadsheet that did it, which then expanded into some other things. * Spreadsheet here, which has four tabs based on different views on how best to pick the fair place to bet where you and someone else disagree. (The fourth tab I didn't make at all, it was added by someone (Luke Sabor) who was passionate about the standard deviation method!)  * People have different beliefs / intuitions about what's fair! * An alternative to the mean probability would be to use the product of the odds ratios. Then if one person thinks .9 and the other .99, the "fair bet" will have implied probability more than .945. *  The problem with using Geometric mean can be highlighted if player 1 estimates 0.99 and player 2 estimates 0.01. This would actually lead player 2 to contribute ~90% of the bet for an EV of 0.09, while player 1 contributes ~10% for an EV of 0.89. I don't like that bet. In this case, mean prob and Z-score mean both agree at 50% contribution and equal EVs. * "The tradeoff here is that using Mean Prob gives equal expected values (see underlined bit), but I don't feel it accurately reflects "put your money where your mouth is". If you're 100 times more confident than the other player, you should be willing to put up 100 times more money. In the Mean prob case, me being 100 times more confident only leads me to put up 20 times the amount of money, even though expected values are more equal." * Then I ended up making an explainer video because I was excited about it   Other spreadsheets I've seen in the space: * Brier score betting (a fifth way to figure out the correct bet ratio!) * Posterior Forecast Calculator * Inferring Probabilities from PredictIt Prices These three all by William Kiely. Does anyone else know of any? Or want to argue for one method over another?

Hi all!  Nice to see that there is now a sub-forum dedicated to Forecasting, this seems like a good place to ask what might be a silly question.   I am doing some work on integrating forecasting with government decision making.  There are several roadblocks to this, but one of them is generating good questions (See Rigor-Relevance trade-off among other things).   One way to avoid this might be to simple ask questions about the targets the government has already set for itself, a lot of these are formulated in a SMART [1] way and are thus pretty forecastable. Forecasts on whether the government will reach its target also seem like they will be immediately actionable for decision makers.  This seemed like a decent strategy to me, but I think I have not seen them mentioned very often. So my question is simple: Is there some sort of major problem here I am overlooking?  The one major problem I could think of is that there might be an incentive for a sort of circular reasoning: If forecasters in aggregate think that the government might not be on its way to achieve a certain target then the gov might announce new policy to remedy the situation. Smart Forecasters might see this coming and start their initial forecast higher.  I think you can balance this by having forecasters forecast on intermediate targets as well.  For example: Most countries have international obligations to reduce their CO2 emissions by X% by 2030, instead of just forecasting the 2030 target you could forecasts on all the intermediate years as well.    1. ^ SMART stands for: Specific, Measurable, Assignable, Realistic, Time-related - See  https://en.wikipedia.org/wiki/SMART_criteria