Trusting Experts

Gregory: Probably what we are going to try and do is, I'm going to explain to you why I'm right and then... I'm joking. I'm going to probably set up my stool for the discussion about whether or not to defer when someone's come in the community to discuss epistemic modesty. I guess Ollie will then...

Oliver: Bring my perspective into it, contrast it a bit, and that will be a starting point for the whole discussion.

Gregory: By way of introduction, there's many cases where deferring to what other people think, or people are who are maybe more informed than you think, is pretty inarguable and pretty obvious. I think that's common knowledge amongst most people here. A common example given is, you know, if you're taking your bins out or taking the trash out on a particular day of a week and you see none of your neighbors have left their trash out on the same day of a week, you probably think like, "I've made a mistake here" rather than "everyone else's made a mistake." And then change your mind because of that. And likely if you read a textbook and you see something in that it looks a little bit surprising to you it's probably more likely that you just don't understand it rather than the textbook makes a mistake. That doesn't mean textbooks are infallible, but I mean you should probably like... if you have to guess which one's right, you or the textbook, you should probably guess the textbook in most cases.

So that's all the really inarguable stuff, which I think Ollie is going to be arguing against very vociferously for the next hour. Alas for me. But where it gets a little more complicated, there's lots of areas where we do often take ourselves or license ourselves to disagree. So like in politics, in various complicated factual matters, in religion and all sorts of other things besides. And there we think, even if they're experts, we still think we are within our rights epistemically to go actually these guys are wrong and we're right. I guess my view on this is like, I'm probably like a modesty radical, immodestly modest, you might say, insofar as I think, in essence, what you should always do in each possible condition of things is you should hew your own view towards an idealized expert consensus. So you weigh up different expert classes, work out which ones are more relevant, and do all of that stuff. And then you get an answer based on that.

And that's what you should think is actually true. So your credence should be driven by this, and your own personal impression of where the balance of reason lies counts in most cases for virtually nothing. It counts no more than another person similarly well informed as you, coming to the view they have. And so that typically means in my day to day life, I end up sort of deferring quite a lot. And so, I don't have a particularly strong political views or like, hot button topics because the fact there is controversy means I probably shouldn't be very confident that one side is right or the other and so on and so forth. I guess the EA relevance of this is in many cases, EAs do take views which like outside side or often against expert consensus, and how often is that a reasonable thing to do or not reasonable? And obviously given my own ideology here, I tend to be leaning in favor of thinking that the EA community should be a lot more cautious and more deferential to existing bodies of expertise. So that's like me as it were setting out my stool. I look forward to Ollie pelting me with rotten fruit, as appropriate. I'll hand over to him for a bit.

Oliver: I think ultimately a lot of what Greg said is right. I think there is an important sense in which it is obviously really important to take in due account other people's epistemic states and other people's observations, and their current opinions in forming your own opinions. But I think there are various problems with trying to do this naively. I think the perspective that I'm taking, is not necessarily have something deeply flawed with something like the complete, philosophically rigorous view of what Greg is saying, but more something where I'm saying that if you do naively the thing that Greg recommends people do, you will end up running into various problems, and I think these problems are kind of illustrated by three of the topics we hope to cover today. There is the question of who are you going to count as an expert, which is I think is underlying just if I'm going to go by expert consensus there is this key question of, who I will let trust as an expert? In questions of religion should I trust theologians? They seem to have thought most about it. Does mean that God definitely exists given that most theologians seem to think God exists? But then we are like, ah, maybe philosophers. So we have this interesting question of who do we trust as our experts?

Similar, a related question is when are you yourself an expert? At what point do you cross the threshold to think that you are the world expert on a relevant topic? Maybe you are the person who has done the most research in the world of all the people that exist into a certain form of quantum mechanics or in certain questions about online discussion platform design (which I hope I have achieved). And in those situations, I think that is just a question where I expect where if you do the modest thing, and if you kind of carry forward the attitude of modesty into that situation, you will usually make bad decisions.

And third of all, there are important second order effects, where if you have a group of people who are generally primarily making decisions based on other people's opinions, you run into many, many information cascade problems that we often see in markets, often see in various real life situations, panics, where everyone else is primarily paying attention to what everyone else does. And then you get stuff like bystander effect, you get stampede and stuff like that. And that is a problem that I think can be avoided with very rigorous and careful thinking about how other people came to their opinions. But it's something that if you're going to naively take the modesty stance, I think you're going to be on average doing worst than I expect most people in this room to be doing. Okay. I think those are my three big perspectives and objections that I have with the broader framework because I think philosophically, I agree with a lot of things that Greg says, but I think the question is more about, how are we going to actually implement these insights about epistemology and understanding.

Gregory: Can we go with the first things first, maybe? So the question of what counts as expertise. Right? And so I often, in fact, I've written at length, at great length on modesty online. My idea is, I often like to use is that maybe somewhat loosely, subject matter expertise is basically the same thing as expertise per se. So to answer "is it good to have a death penalty" look for people who work in criminal justice, is a good to increase or decrease minimum wage, maybe we should talk to economists, I say as one enters the room just now. If you want to know if you're sick or not maybe consult a physician. I think there were many cases where people would agree that's a useful first pass heuristic.

But there's a key question as to, in many cases it seems like this often co-locates with social status. So I used to be a doctor, doctors have this quite good reputation, present company notwithstanding, and that means that you're taken maybe more seriously than you deserve to be. There are like lots of stories actually in medicine going further where people are overly deferential to doctors who are doing really stupid things. Or like the nurse, who is not in the same high status profession the doctor is, sort of let people like me make terrible mistakes and cause damage. So that's obviously like where it can go wrong and definitely what I would want to avoid.

The question is we have some data, some interesting data Ollie wanted to cover. So I'm stealing his thunder, on where very simple scoring rules can beat commercial expertise. So there was like reason rules from Kahneman, there's prediction markets and all of this stuff. And so it seems you don't necessarily need lots of subject matter knowledge, beyond being an academic, being a professor, to actually get better answers than the supposed expert class. Do you want to elaborate more on that, so I'm not making your point before you can do an even better job of what I'm attempting to do.

Oliver: I think like one of the most interesting questions that illustrates this most is the super forecasting literature, where you have this interesting thing where you have the super forecasters trying to make judgments about global politics, trying to make judgments about technological advances. And those are people without subject matter expertise. They are people who usually are generally well read, and can do Wikipedia searches, and then just try to apply kind of good forms of... good rules of reasoning, that are pretty domain general, like they don't really depend on a specific domain. And in the super forecasting literature, that has shown to very often outperform experts. And now you have this problem where like okay we now... we have the experts are forecasting, which we can think of as kind of the experts at answering certain forms of predictive questions. And we have domain experts who are hopefully good at answering domain expertise questions. But when domains widows intersect, which is quite a bit obviously, we are kind of stuck if we try to do naive expert aggregation.

Gregory: I think it's worth... maybe just on the second point. So it's obviously the case that if you're trying to defer in the epistemic sense, your criteria for expertise is just like predictive accuracy. So in a world where the super forecasters and intelligent analysts are predicting something and you know the super forecaster has got a better track record, even if the analysts have a higher social status, the modest thing is to go with the super forecasters and not with the intelligence analysts. The problem is, of course, that we may find, which we may go on to later, is that maybe the general skills required to make good predictions about various events is like this single key skill, and doesn't really have very much to do with particular domains of expert knowledge, and so it shouldn't be like deferring very much to these sorts of people.

I guess my own view, to like till my hand preemptively, is that although it may be the case for your super forecasting, you might be able to sort of take issue with intelligence services or similar things like that, if you're a typical person in the audience, you're probably not a super forecaster. And so even if there's even better people out there, you should be deferring to, instead of an expert class, a typical subject matter expert has got advantages over you, or so I would hope, but that may lead us into the next thing a little bit too early. On which, more later.

I guess the other thing say, that Ollie alluded to, it is a lot harder than it sounds just to defer to the experts because, taking your example further, like if you ask theologians whether God exists, they all agree. If you ask people in general, most people believe God exists, worldwide. That's so-called the common consent argument for God's existence. We also see, but then you go, yeah, fine, but maybe smart people are expert class on this. So you stratify by IQ. People with higher IQs trend atheist compared to general population. Okay. That might be something. But wait a minute, Well, what about philosophers? Philosophers themselves also, philosophers generally also trend atheist. Philosophers of religion, though, who spend lots of time thinking about whether God does or doesn't exist, they actually trend strongly theistic, at least in the US.

And then you go, well, who should we undercut? You go to underlying features. Maybe like people are really just naturally religious gravitate to philosophy of religion. They weren't persuaded by the arguments, and you can have lots of very complicated discussions over who counts as an expert, and like make a rod for my own back. There's a worry whereby you can liberally gerrymander who you count as an expert to get the answer you wanted to have in the first place. So, weigh these people slightly more highly, and there's lots of fudge factor, so you can twist the dials to, to get the answer you'd like. Obviously, epistemically virtuous people like me would never do anything like that, but you know, better off for the rest of you guys. So whatever. It's harder, it is very hard to do. I would still insist or suggest, I'll suggest it can still be better than trying to strike out on your own and try and form your own impressions of it, which is perhaps where we begin to route back into a topic of disagreements.

Oliver: Yes. So I think one area that I'm curious about kind of exploring, that I think could be really relevant here, is I expect us to have some disagreements about to what degree do we think there are a set of key underlying, think of them as general rationality skills, or you could think of them as general philosophy skills or something like that. Where you can easily imagine a world, and I like to always... there's a straw man mathematician perspective where you study mathematics, and mathematics has like eight branches. You'll study algebra, you'll study linear algebra, you study some of the more graph theoretic stuff, and now you really have eight models of the world and everything is really just an application of this. Like after you notice eight models, like voting theory, no problem, you just apply some graphs to it. Database design, no problem, you just apply some algebraic constructions to it. And so, there's this idea that we might be living in a world where there are very few simple models that we can generally apply to a lot of different things, and that ultimately expertise in those models dominates the ability to predict accurately in that domain over this specific subject expertise.

I don't think this is truly as extreme as the naive mathematician's view suggested, but I think it has quite a lot of merit in general, which we see both in super forecasting domain, but also in the domain of thinking rigorously and reasoning rigorously, which isn't necessarily only as an are, a super forecasting domain. I think in many ways I would describe it as something like philosophical competence, where you have a concrete example.

One of the big problems that I heard Open Phil has often run into, in their discussions with various experts, is they try to make a decision of like how likely are various risks, and they go to the relevant experts and they asked them, “how likely are those risks?” And they answer them with “well very high.” And you're like, “well what do you mean with very high?” "Well, like definitely high." And you're like "is it 50 percent or 70 percent?" It's like, “I don't know. I can't really put a number on it. High.” And you end up kind of in this situation where, I think there's an important sense in which general skills about how you interact with your beliefs, and even being able to make precise statements about the world that are general in the sense, like, could screen off just the sense of domain expertise. And I expected we probably disagree about the degree to which there is specific domain expertise in detail to these different areas.

Gregory: Like can I just reach the summit of the epistemic function with all creation arrayed before you, is the only way you can make good progress. So that's like, yes. So I think you do anticipate correctly. I'd be like, "well I'm not entirely sure about that." I mean, it seems the case that there's like something which super forecasters have which involves doing lots of different things. They have to be able to predict. There are specialist politics forecasters, there's some of that. They often look quite good in various domains, so definitely the transfer capability set, it's worth saying that they have some background knowledge. They're not usually doing things literally from nothing but it's like not very much, like something like a reasonably low time level expertise, rather than spend like two decades studying like dynamics of politics in a certain area.

I guess there are also other areas you want to look to besides super forecasting, because like you could say there's like civilizational inadequacy, whereby people aren't trying to protect certain very important events. And so maybe experts haven't really like been trying to develop expertise yet in that area. Well it's like expert performance. It may be the case that it's quite widely separated, so people who are really good at Go on really good at Chess necessarily, or like Poker, and things like that. And we have pretty good evidence that actually, when they can play each other. There's lots of things you'd like to learn, to become really good at chess, which aren't just general skills. Things like IQ and stuff help, but it helps to the degree you'd expect. So I guess my view is that lots of bits of reality have this property whereby you have to like pile in lots and lots of precise knowledge to make good progress in an area. We might just be like at a very early stage of forecasting. It's like in a field where you can become an expert in. Maybe the forthcoming and, one hopes, Tetlock-infication of governments will lead more in that area but who knows? Maybe I'm wrong.

Oliver: Yes. So whenever we can find maybe a set of predictions or something like that, where we completely expect different groups of people to give different quality of answers. That would help us maybe tease out some of the things.

Gregory: Yeah.

Oliver: So I wonder if we could find the edge of like where I expect super forecasters to stop being good. Where just like you expect super forecasters to stop being good, where maybe it works, where like I can imagine that there is a world where you... Like let's say we go into the domain of quantum physics or something like that, and where I think I still need to reflect a bit more about it, but naively would expect that, I think like my trust in super forecasters would extend further into domain of like narrow quantum physics, and getting questions such as like, "Is cold fusion possible?" Something you talked about it in one of your blog posts, and we ask physicists, maybe even a pretty good physicist from a top university, and we ask a super forecaster, and who would we trust on that? And I think I would be pretty torn. I really do. I think I would have a good chance, especially if I have a super forecaster work on it for a week, I would have a pretty good chance that I would trust them more than the average physicists. At least PhD students.

Gregory: I definitely agree with the direction of where we differ. I'm just not quite sure if we'd like can crispy like, find out exactly where, because not necessarily, if you have a fairly wacky thing like cold fusion, physicists wouldn't necessarily usually work on cold fusion. Or have any knowledge of it, like they'd just think it seems weird to me. It's effectively outside my... I agree in the general sense that obviously, you'd want several key core skills, like having good accuracy of beliefs. You would expect these things to apply more broadly than I would. If you're trying to predict whether the answer is like Ramsey 55, or Ramsey 77, I would be much more willing to trust my intuitions about super forecasters to give more evidence for that, or similar things like that.

It would generally be nice, I guess... One interesting area perhaps is like, data on rationality training, like calibration training. There's not really crisp evidence of improving like job performance is my understanding. I'm not very acquainted with this literature, but my standard understanding is, we do this stuff. They learn these cognitive biases, and yet you haven't gotten better. Now it could be a skill like this doesn't transfer. If you just drill it you might become smarter, but it could be something like, it's not really training the underlying skill. But if it was improving the underlying skill and yet you don't see improvement in performance where we think expertise would matter, we would give different predictions about how likely that is to be the case. But again this might be a little bit too woolly to really pin down.

Oliver: I actually think this is a very important domain, and I think one of the things that kind of underlines a lot of my edifice of opinion here is, there is this interesting question, where at some point in the 17th century, humanity started being able to solve problems that it was never be able to solve before, remotely. Like we started out like being able to kind of, if we really tried, build a simple city, and then 200 years later or 300 years later we build rockets that flew to the moon. We built computers, we built the Internet, and it really seems like something... like, we see something like the Flynn effect. You can imagine maybe something like general competence increased and people got more intelligent. People had less like nutritional deficiencies and maybe that's the reason why we can do all of those things.

But I think the case for that is not strong enough. I think ultimately there is a strong argument for something like, there was a quantification and mathematification of a lot of these fields of science, that allows us to build the methods of science, build out physics as a domain, and then start just solving lots and lots of problems with this pretty small set of tools. And like, if you look at all of human knowledge at the time, this at the time, if you look at the 17th century, like what you would study at university, it was primarily rhetoric. It was primarily like lots and lots of theology. And then you would maybe spend something like 10% of your time studying mathematics, maybe 15% of your time studying mathematics. But that branch turned out, that kind of relatively small aspect of human knowledge, turned out to be very widely applicable, and to solve a really large domain of problem. And I think this highlights an important sentence in which we have, I think we have an answer to discretion of whether there are domain general skills that allow you to get better at solving problems in a large range. And I think the answer is, a lot of like modern quantified science and the evidence we have for that is the scientific and industrial revolution. And I wonder if like...

Gregory: I think there's part of the point I agree with, but I'm not sure how much of the variance it explains in the classical challenge, here. I mean like inferring, I mean like, there's a perennial problem in the field of like, macro-historical judgment is very hard. Too early to tell what caused the scientific revolution. It might be too early to say we're confident with the scientific revolution was because we discovered a maths, when there was like lots of precursors. Now we discovered calculus reasoning just before, and I think it's quite hard to work out things like intellectual traditions. But I agree in the general sense. We have like sort of really crisp historical data showing we made this like advance in thinking tools, and this great wealth of understanding opened up in front of us.

I think that's all very favorable evidence. It's like been, to a very large degree, these very key skills you can apply to lots of stuff, rather than my view where it's like, you have like lots of like hedgehogs. Hedgehogs or foxes? I think hedgehogs. You have to burrow you way into your area to like, get good answers. I'm just wondering. We might be thinking of moving to point B? You get the last word in, but...

Oliver: No, I think that's pretty natural stopping point here. Okay, cool. So next thing that we wanted to cover is when are you an expert, and it covers a lot of very similar ground. I wonder whether you have like some initial thoughts.

Gregory: Yes. I think like where we can go next is we can, we can discuss often, in the general sense, like how the different classes of cognizers are like, and how much there's like a single axis of thinking skills. Like with people who were top, who are able to get lots of things, or if it's like very, very dimensional. Like, if you're an expert at math, you can make good predictions at math. If you're an expert in politics, you can make good predictions in politics. I know there's lots of different things and therefore if you're like, but then what do you do, and if you're a typical person who is not an expert in many things, or in my case an expert in anything, what you can do instead is say, if I'm like pretty mediocre at everything, I can steal the knowledge out people have got, and steal their predictions and use that instead. And that just seems like a strategy-stealing, superior approach. And then I guess I'd then say even if we... even if we disagree or how much there's like, one key skill, I'll say you can often maybe like surpass what's a typical individual in this community can do, if we defer more, rather than saying I've read, I've looked into stuff and rationality, I've read various works on like how to by rational. I see myself equipped and have very confident pronouncements to make, on let's say macro economic theory or public health and stuff like that. And that's the typical principle of mediocrity rather than modestly. The typical person, even in EA, who are very clever people, of course you all are, generally would always almost always do better if they're deferring to someone. Whether that's a super forecaster or a domain expert is unclear. But I do think we are unexceptional cognizers in this whole sense, or at least not so exceptional. We can always bank on ourselves to want to stake out our own views constantly, but maybe that's not quite getting to the crux of the matter, as you might say.

Oliver: So I think this is... I feel like this is almost responding to your previous point and I kind of wanted pull on this, but it's kind of something that's been... where I have this thing. I have this feeling that we both agree that the question of who is an expert and how should we aggregate opinion, and something like when exactly should we believe someone else's opinion and how should we integrate with us? Like if we both agree this is a very rich and important domain, in a sense that it's important that we figure out the general rules of this space, we figure out how to aggregate knowledge like that. But I feel... I don't know what it is, but I have sometimes the feeling that when you advocate for modesty, that you advocate not for the study of a rich field.

I feel you're not advocating necessarily being like, something that... I think the thing that I would be definitely deeply behind is something like: I think the question of how to aggregate opinion of the people around you is important, and the answer is not obvious. Like the answer is, here are some interesting considerations that we have. And then you would like maybe need examples, and then somebody would want to start getting expertise in this domain, meta-expertise. That would be, let's look at the example of super forecasters with physicists, analyze the structure of that, and then come to some conclusions, and then maybe try to extract some general principles from that. And then I would be like, yep, those are good principles. But like kind of making it clear that there is an art and a skill here that needs to be built up over time, where I'm worried that when you usually say modesty, you're like just, be less arrogant or something like that.

And I agree that maybe some people have like an aura of arrogance, but it's like a very... something like trying to tell a physicist that they're not making a lot of progress, because they always write too large in their notebook, and so they run out of space before they finished thinking on a current notebook page. And I'm like, I guess I can see how this is maybe an error that someone is making, but it really doesn't seem like the most important dimension, or something like that. Like arrogance or something feels almost like, it's a relevant variable, that I think is important to calibrate, but it doesn't strike me as one of the key variables. And I wonder whether you might disagree with that.

Gregory: I didn't know. It may be a little bit hard to litigate the examples I have in mind, which may not be very helpful to discuss. I think my view is, if maybe the case that it's not necessarily straightforward, that doing like the modesty thing well is in itself a bit of a skill. But I think it is also the case that maybe I can't identify really clear mistakes. So there's the case I got from a friend where someone said by way I've disproved the continuum hypothesis: I've god this notebook, would you like to have a look, and it just feels like in that sort of world, we answer, no, I can just tell you with probability 1 minus epsilon you're wrong about this, and you should have thought the same because there's lots of proofs of this, and the rest of it.

Oliver: Yes.

Gregory: So I see like I definitely feel like there's like steps you make to do better. I often see like sort of like really obvious mistakes, and then the simple word, is please stop being so epistemically arrogant. But let me say what you do instead is a bit more involved. But as a crude rule of thumb, I often find this seems to do better is I guess my view. But also the other thing whereby like epistemic immodesty, and immodesty in general are not exactly the same thing, and they can co-locate. You might get annoyed or people being like a bit cocky. So for me, I'm like cocky all the time, and I'm like preaching epistemic modesty, so these things do come apart in various respects. But in the same way, it's important to stress that one shouldn't poison one's epistemics by personal dislike you have for a certain communication style, or anything like that. I may not always practice what I preach here, but I do at least recognize I may be at fault.

Oliver: Yeah. I do think this is something that I probably expect to be useful to clarify. Like we both agree and we kind of had an earlier conversation a bit about this, where there's just this really important distinction between what we think of as social expertise, where there's like, I guess the degree to which our gut judgment says that somebody is an expert, and something like the actual epistemic expertise. The actual track record of how much will they reliably get questions in this domain right. And just that those are definitely two different things, in different domains, that can get completely different answers.

Gregory: What made how completely might be another...

Oliver: Yes, that might be the-

Gregory: I might be more deferential. It's like sort of cultural expertise. Like okay, if I've got an expertise it's good. If you're an expert in this area, even if you're not a super forecaster, you might still be worth deferring to in many cases. Probably like obviously that disagreement over how well we can correlate may not be a large part of a wider discussion I guess.