Science and Prediction Markets Professor Robin Hanson https://desci.nyc/transcripts/science-and-prediction-markets Automatically generated transcript. May contain errors. [0:11] better uh move the light out right right right yeah yeah works we almost lost our speaker okay um all right thank you all for coming out if you guys in the back um feel free to way um okay right so thank you guys all for coming out I mean it's a Sunday this is our first time who likes coming on a Sunday and like Monday or this is New [0:57] York this is New York okay cool um um okay so um this is D NYC and it's basically like a monthly is Meetup for people that are interested in science in New York City I thought there was like a lot of run clubs and there was a lot of drinking things and there was like a lot of like stuff like that but there wasn't like an intellectual space where people could uh learn about science and sort of continue on their education so we had this idea to uh like host sort of monthly is speakers uh around science topics that were novel and interesting and we really try and get uh speakers that are somewhat [1:38] controversial or somewhat like doing something that's a little outside of the normal so the ideal speaker like half the people should love and half the people should hate it's like the joke should everyone shouldn't laugh at the joke um okay so um it really helps us um if you guys if you have we we got our Twitter account back sort of which is which is nice um so if you want to follow our new Twitter account you can stay up to date with um new events that we have and um pictures after the event airdrop opportunities um we also have our Instagram which we're sort of still trying to like if anyone here wants to [2:19] work on like a social media manager or something with us uh send us a DM uh yeah send us a DM after and you can do it or Ray can you put in the jobs the jobs yeah in the last life jobs Stadium science stadium. science jobs um and then you can follow us on Instagram and if you tag if you tag us like that's helpful too so take a picture of this follow us I see nobody taking out their phon so so take a picture of it um uh tag us take a picture if you're having fun if you're enjoying it you should tag us uh invite your friends that sort of thing [3:00] okay so um we'd like to thank our um sponsors um so this is like a really important one um binance is sponsoring this event binance is amazing they uh they were launching Stadium science which I'll talk about on them um they have this new thing called Greenfield which is an incredible product um to store hot data which we'll also be using which we'll talk about in a little bit um Stadium science is sponsoring this dite NYC we thank them um we also got sponsorship for movement Labs they're very fast um L1 um based on the new technology we thank them for their sponsorship um Bas is also sponsoring we [3:41] have a couple people here from Bas Thank You for Your sponsorship as well um we appreciate it and they're also sponsoring all the pizza which we appreciate as well I think we had like $750 worth of pizza for you guys and I think you guys destroyed all of it so congratulations on that um lastly Arch lending we thank them they're the ones who provided this beautiful space um they have this cool logo if you need to take out a uh several million dollar loan at some point definitely consider uh take using Arch lending okay so now we get into what D is this was the one we did with Brian Johnson we had a thousand people there [4:21] um it was an incredible event we've had other speak speakers here um we've had Michael Levan we've had who talked about bio electricity we had Martin scy talking about AI we've had um all sorts of people Aaron Doza talking about the Olympics that were with performance enhancing drugs we had people from talking about politics and science so it's a really wide range uh if you have interesting speakers that you are interested in um send me a DM if they're controversial if they're interesting uh that's the best type of thing um speaking of new events we have an event actually tomorrow because of the wers of scheduling um it'll on water bears which will be incredible um you [5:02] definitely come there it's sponsored by the salana foundation it's sponsored by the water Bear Foundation that's sponsored by scub these are amazing organizations which we appreciate um and you're going to learn these T tardigrades or water bears are one of the most incredible creatures they go 20 years without food they can live in outer space they're one of the only animals that can live in outer space um they are immune to high amounts of radiation um and they're basically like walking heads so I think they're actually like the perfect example for what decentralized science is something that's sort of indestructible and that does not care don't care don't care water bear does not care and then we have an if you want to rspp for that um you can scan [5:44] this QR code here give you guys a second to scan the QR code is that in Salon Foundation like a office yes it's at the salana foundation office in Lower East in Lower East Side please do c me if for want to it'll be a fun it'll be we have a couple surprises in store and the air drop maybe an air drop you never know water bears aren't too controversial so oh they look at this this is controversy um we have another talk coming up um when AI eats science uh SJ is an amazing guy he's really smart uh he has he's the has the fifth number he has the fifth most um followed um GitHub account he's a YouTube things with [6:25] millions of followers he's really done a lot to democratize science democratize AI democratized programming super cool guy he's going to be talking about how to do science like what science looks like with like o uh o1 and like Advanced models like that and trying to like figure out how to use these new types of advanced models to to do science and I think he's really on to something I in my personal opinion uh I think that's how science will be done in the not near future and uh it's any sort of White Collar cognizant job is going to be very um well done by an AI that can just churn through it at a faster rate than a human can so that will be on December 15th um that'll be an amazing talk so if you're interested scan the QR code and you can [7:07] join us for that okay so how do people know or know what decentralized science is okay how many people are in science cool okay so of the people that are in science like I'll Just Popcorn around what what things are good about it and what things are bad about it I'm kind of curious um funding funding is is bad or is bad oh yeah it's very hard to get okay anyone else it's expensive it's expensive okay so it's all money okay yeah I think it's really over regulated over regulated I think it's rigid rigid okay what about the accuracy of it does [7:47] anyone think that the accuracy of traditional science is perfect or not it is perfect or it's not perfect it's in the toilet it's in the toilet okay so maybe it's not just only the money but it's ALS I the incentives the incentives are okay so there's like a lot of things that maybe could be could be broken that broken with traditional Academia um partially having to do with the incentives or how it is easy to get funding or um I also think the uh the accuracy and you know traditional science is thought to be so accurate and it's the gold standard and there's peer review and there's can be nothing wrong with it I think that's another thing that uh people in Academia might have uh second guess a little bit and it's [8:27] expensive oh you guys were just reading out the slid okay well okay whatever um so I'll tell you a little bit like what we're working on um I started a company it's called Stadium science we're trying to sort of democratize science using um us making science more fun as a primary thing and then changing the incentives on how it's done so uh we want we raised money from polychain and DOW five um and we're looking to hire more people if you're interested um the way it works is um you can like you can like produce data sets using wearable data so your aura ring or on your sleep data or stuff like that and you can um track each other and see [9:09] how well like each person is like sleeping but then you can also sort of begin to make bets on the each individual person's uh data that they're producing so you you link your aura ring to this site and then you share you share it with friends and then um you do prediction markets on top of this data set in a very similar way to you do it with poly Market or something so um you can see you guys can all go to stadium. science you can track all my sleep metrics it's all publicly available so you know when I'm sleeping well and you know when I'm sleeping badly and you can sort of bet on if you think I'm going to sleep well or not tomorrow I think this is a cool example of like a Time series data set where [9:50] you're trying to sort of anticipate and predict um the next point in that um based on Insider information that you might have or just based on my Trends in the past and I think like at a at a high level like science is about creating a model of the universe and coming up with the next accurate prediction of the next data point so sort of using this new incentive model to encourage people to predict the next data point could be a very cool thing so you can also speculate on the drugs that I'm taking so if I'm wearing a sleeping mask or I'm taking caffeine each of these is a token that can also be bought and sold um to allow you to speculate on if melatonin actually helps my sleep or not so there's a lot of different ways in which you can participate in uh seeing like [10:32] what's working and what's not working and tracking it with my with actual real world data and then lastly you can bet on it so you can bet if I'm going to be getting more or less than seven hours of sleep and I think this is a cool early example and we We'll add more things in the future okay um and then if you want to download it we have the Apple we have the Android version here you can do it we have a couple thousand downloads it's very cool three oh got some people taking okay nice maybe just stalker is looking at my [Laughter] sleeve okay [11:14] um oh okay okay then that's all any questions on decentralized science D NYC or anything like that 3 two 1 all right well we hope to see you all tomorrow and then lastly um without further Ado we would like to introduce Robin Hansen um to speak on prediction markets and Science and Academia and uh Robin is an incredible person I've been following him uh not his sleep but his blog should all check out um I'm sure he'll promote it at the end but it has a lot of really insightful things on incentives and economics and you should [11:54] follow him on Twitter as well uh I really enjoy his tweets they're all across the board and very insightful uh every one of them so I I appreciate that and encourage you guys all to follow him and uh lastly I think that's really cool is you know vitalic um you guys know vitalic course he's he's I think he it would be correct to say that he's like the inspiration for a lot of vitalik's blog posts and stuff so yeah he's the person who cited in a lot of the blog posts and his thought uh and I think is is a really cool thing and um without further Ado I'd like to welcome hello everyone I'm a professor so I'm used to filling a big room with sound no one in a need to do that [12:36] here uh I started out in physics a long time ago and my physics professors kept telling him about how great science was and science was the thing to credit for all the cool physics I was learning and I kept hearing that I kept thinking what what is this science thing and so I decided to get a graduate degree to go into a PhD program in philosophy of science in science studies at University of Chicago to figure out what the hell this science thing was and relatively soon I realized there is no such thing there is no like scientific method or like simple way to describe what it is it's a community of people who do things but I read about the history of [13:18] Science and sociology of Science and philosophy of science and then I went back into physics and then I did AI for n years and then I wanted to go back into school and over all this time I I was thinking about science and Academia and its problems so I left Academia in 1984 40 years ago to go to Silicon Valley because two things were happening there a previous AI boom was happening and then there was this thing called hypertext publishing which was the vision of the web that was coming and people had these idealistic visions of how the web would help cure Academia and policy discussions and I bought into [13:59] into that and then I was thinking about well was that really going to work and then because they were all kind of rabid Libertarians the idea was available to me to think of betting markets as an alternative solution so then I started to think and WR about that around ' 89 and then I was planning going back to school and I for I actually applied to an and that was accepted by and then I accepted going to a science studies program and then a week after the weekend I said no maybe not and I finally did a PHD in social science science at Caltech but I'm trying to tell you that this issue of what's wrong with Academia and science has been a focus of my life for a very long time and once I saw prediction Market a [14:41] solution I was excited and I gave a lot of talks and then I decided I needed to go back to school to get contacts credentials because otherwise I wasn't going to get anywhere and so I went back to school in 93 to get my PhD which is still a long time ago I get it uh and then I've gone through through a sequence of stages in realizing that my initial visions of how prediction markets help really weren't going to work and so I'm trying to show you that I've tried a lot of ideas in my lifetime and a lot of them I've realized wouldn't really work because the problem is really hard and I'm going to impress you now with how the problem is but I'm still going to give you what I think is a solution but it's a solution that's gone through a lot of iterations a lot of things have been rejected is the [15:22] point so let me tell you a bit about the history of science Academia four centuries ago the science Revolution was getting started and for the next two centuries the main funing of science happened through prizes which is now something people hope to revive because it seems like to an economist a more effective way to fund research you only pay for the results when people actually produce results you want uh but around the 1830s through fraud and um basically fraud both the academics in England and France overthrew the up until then [16:04] common prize system for funding science and replaced it with grants the scientific Society for promotion of Science in Britain had been collecting dues on the on the claim that they would fund prizes and then they decided no we're just going to fund grants and then for the next 30 years they only gave grants to the same 10 people over and over again IE the same 10 insiders and in France they had a long history of giving out science prizes and a very rich person monu I think something like that had given a big prize and then the scientific Society leaders went to the king and said this guy gave money in his will to hand out his prize but we want to use that for Grants instead please King just let us take his money and use it for a different purpose than we promised and [16:44] the King said okay and then from then on you see grants became the more common thing in science rather than prizes even though they're less effective at promoting research and then around 1900 we had the rise of both academic tenure and peerreview which are both ways for academics to take control and rest control of their lives from Outsiders they see and to sort of be self-sufficiently be able to decide their own faith and so that's pretty much what any profession wants right in some sense The Insider top people in a profession want those outside customers and suppliers or whatever to just shut [17:24] up and go go away and just listen to us and do what we say and give us the money and not don't ask anything for it and just like let us continue to do whatever we want but academics have been remarkably successful at achieving that ideal compared to most other people in the world right so there are say prestigious Cooks in the world but prestigious Cooks just don't make us all eat whatever food they want and and decide it's good we have our own judgment about what food we like and so we sometimes listen to prestigious cooks or go to their places but they have limited influence but but when we don't think we can judge what's good we often give deference to experts [18:04] like academics and then they have enormous autonomy and then they can just do whatever they want and that's basically the situation it's largely true of medicine and law but especially true of Academia that basically it's a mutual admiration Society of the top academics can just decide what is good Academia academics and then who are the good people and then you know they they want us to give them their money and other sorts of supports but they don't want us to have anything to say in terms of discipline of what they do um and so in history the sort of the people who could say something were the funders people who gave them money a [18:44] students maybe who would choose where to study there would be uh you know there would be journals say who might decide what to publish or there would be uh a journalists who might decide who to interview and over time all those customers of Academia have been bullied into the attitude of you don't know anything about what we know everything and you should just believe whatever we tell you about whatever we want to tell you you know that has become the sort of winning position that's really amazing if you think about it think about all the other professions of the world that would love to be in a position like that but they managed to achieve it in part because they decide other prestiges in [19:26] the world they because they run universities Etc they're the ones who decide who else is considered prestigious in all the other professions and that's really a powerful position to be in all right so my first vision of prediction markets was if only we could just make a market sitting next to each scientific topic that was controversial then the betting markets would have a better consensus and that would force the academics to listen to the betting markets and then have a more reasonable consensus but in fact academics just ignore betting markets unless so they often make one-on-one Bets with each other but they only do it on equal status and they don't really set up prices that would disagree with them and so honestly that doesn't work having the main customer [20:08] betting markets just be people who want to bet for fun just doesn't impact the world of Academia so the key idea Academia is this world is very competitive it's very entrenched they've worked for a long time to sort of push away people who might try to control them and uh our challenge is to somehow get them to even so listen to us about what we want out of Academia what we want out of science that's our challenge so we might have thought just having lots of ordinary people betting would be do enough that's just not going to do it the next thing that occurs to people is maybe what if we get could get the funders the funders might say we reason with the funders is say look prizes are more effective for you they will get [20:50] more acade more research progress don't that's what you want right so why don't you give money in the form of prizes and of course what we'll find out is they are just a player in the game like like everybody else what they're buying is association with credential depressiveness and they're getting that now from the way they give money and the alternative methods we would propose would get them less of that so they don't want it so and and journal editors are similar things so I was part of a project uh 10 years ago where people were trying to predict which papers would replicate in the replication crisis and finding out prediction Market could predict that and so could other mechanisms and the idea was to go to journalists and say would you tell everybody that you're going to look at this metric of whether a paper replicates as an indicator of whether [21:31] you want to publish it because surely the journalist wouldn't want to publish papers that don't replicate right no they're fine with publishing papers that don't replicate they're another player in the game they're also getting what they want out of the game which is prestige they don't care that much if it replicates or whether it produces progress so you've got a game of people all of whom are buying for Prestige fighting over it gaining Prestige their key customers again are student patrons journalists everybody who gains Prestige by association with their credential impressiveness that's the actual game going on and they are doing a very good job of selling that Prestige and [22:11] managing it and making sure other people can't you know grab it from them but if what we wanted was intellectual progress then it's not doing so good but who the heck wants intellectual progress who is our customer for that exactly is the point how do we reform this process by getting the customer who wants an ual progress somehow to have a voice so I hope I've shown you just what you're up against that is for centuries the trend has been in the other direction we had better institutions before where University presidents people who gave out money journalists students all were more critical and more uh judgmental about Academia and more willing to you [22:52] know reject them or go somewhere else if they weren't getting what they want but today we have all been told no you don't know anything they know at all and you should just believe them about everything so how could we possibly fix this problem I hope I made it seem hard now I will give you my best answer okay here is the lever I think we have we want to move the world we have to find a lever point we have to find a place where we can fix something and push and make something else move what's that fixed Point here's the fix point I would say of Academia today and all through history the thing they won't give up is the claim that the people they most celebrate jobs at Harvard you know [23:33] whatever are the people that in several centuries when people look back will have been the most important people in those areas that's a claim they won't give up on they won't be willing to say well sure this the future people think somebody else around here is important but we're playing this game over here we're fine with our game and you know we don't care about that they won't be willing to say that they will do what it takes I claim to maintain this appearance that the people they are funding that they are celebrating they are publishing they are giving jobs to are in fact the people most likely to be the ones that future historians looking back will say those were the people doing the interesting stuff they claim that now they claim it [24:15] without doing much proof of it but most people accept this claim but I can use this claim as a lever because we could first hire people to do historic studies now of the past centuries ago go back and rank people in the past according to how not only influential they were but how influential they should have been if people should have listened to them and if we can show that we can do that reliably I.E fund different groups to do it in different ways about similar groups of people to show that there is a consistent ranking that comes out of that then we can make Futures markets today in those rankings a century two or [24:55] two in the future that's the vision so if you imagine for every academic of any level even like even grad students I guess we have a Futures Market estimates of their rank as judged by historical analyses in a century or two now these numbers you put up against so when Harvard hires three sociologists you say well your sociologists are ranked number you know 3,000 4,000 and 20,000 on this list why didn't you pick number one two and three and they will have a choice they can either dismiss the market price and say that means nothing that's irrelevant those people don't know anything or they can adjust their [25:37] choices to match the markets or they can go B in the market to move the market estimates to closer to what they think any of those will except the first one actually make Academia more in lined with producing intellectual progress so the the thing that would fail is if they just said all that market doesn't mean anything but but since we have you know growing data that market prices do mean things and we can show track records of the correlation and say you know if you think you know better then shouldn't you come in and bet then so it would obviously be hard to set up millions of markets on millions of people all at once so what you'd want to do is start with some [26:18] areas fund markets in people there ranking them get that to be going and get people to care about it and then try to manipulate it and then move the incentives around in order to create a self-sustaining uh equilibrium in each of these places how do we determine the rankings we would why there so much control so I mean you probably heard of historians of science or people telling you the history of science telling you the story of who in the past was important right I see but that could be Cil attacked is like well we're going to assign historians or people who are going to be acting as historians to go study particular things and then give their ranking and we're going to do that [26:59] for multiple groups you see we're going to show that doing it for multiple independent groups the answers they give are correlated and that will be the the key thing here is that now there might be some ways we do it the answer isn't as correlated so there might be some ways that people just you know get take get paid or whatever to do whatever gets paid but there'll be other ways in which so this is the foundation of say juries right a key Foundation of law is that it's possible to impanel a jury that will actually give you an independent evaluation and of course this is in some sense part of the idea of peerreview right peerreview was just another way of academics taking control from Journal editors like Einstein never got peer reviewed he had Journal letters who said [27:40] that looks like good stuff and published him but peer review is where academics Take Over Control from the journal OU say no no no we want to control that the same way for peer review and funding uh so the idea here is that if there's any one load star that science is anchored around it's the idea that eventually science gets it right we will control it and that can be our lover science eventually gets it right then eventually have a basis for looking back and say who is doing the good stuff and then we can use that Futures markets and those estimates as a way to discipline academ now so that's my vision for how we could in [28:21] fact make it work so you can see how hard this is I have many other ideas that I'm happy to tell you about for how to use prediction markets in other areas of life and those are easier the thing that's so hard here is Academia is at this peak of prestige where they really have control over their world and control over anybody who says anything about it and it's really hard to challenge that and the thing they anchor themselves on is this distant future judgment which is hard to tie current immediate incentives to but this I think would work again the vision is on individual people [29:02] or even individual works you have a Futures Market in how it will be ranked so you the the ranking is important it's not just some abstract number a relative comparison across topics across fields in basically how important it was and look when we look back at the past historian of Science and I've done this in the past you know we can definitely see dead ends we can definitely people were on when we know see that was a key thing that was going somewhere we now know the answer that that was important if you don't think we can see that looking into the past maybe you don't think there is a such thing as [29:43] scientific progress because that would be the anchor on which we have it and there like look there are areas of study or scholarship if you will perhaps areas of moral philosophy where you think there is no such convergence there is no consistent way to look back and judge how good stuff was but in much of what we call science I would think the distinguishing feature is in fact this belief that in the long run we actually figure things out and we can use what we figure out to judge who is on the right track in the past and then prediction markets could predict that all right so uh it's getting hot and humid in here but let's start um [30:23] yeah if you said that uh you told us that they don't care about replication right replication seems at least directly anti-correlated to what will be important in the past so how can we be sure that this would you this will work out they claim they like the appearance that the papers they publish that won't replicate will replicate you see right if you forced everyone to see that they wouldn't replicate they would in fact not be interested in publishing those but they're not going to contribute to helping you see that they're going to resist making that Ober visible right so that's the whole point which is true in many areas like anybody who's out there selling you a crappy product they're [31:04] easily trying to convince you it's a good product but they'd be happy to actually sell you a cheap crappy product than an expensive good product if they can because they'd rather sell you the cheap thing right but if you know that the cheap thing is in fact crappy then you won't buy that and they won't try to sell it to you right so the idea here is to make this visible in a way that's hard to ignore you have yeah I mean I really like where you're coming from I guess what I'm struggling with is um you know universities hire but they don't fund like funding comes from NIH and other mechanisms and there's you know a lot of study around like what they're willing obvious what they're willing to fund their conservatism and like biases like that you know dictates so much of what [31:45] gets so you can through this change like who gets hired but it doesn't necessarily change like what science gets you know fun this mechanism I described would work for funders and jobs and Publications and all that it's not a mechanism specifically for jobs how how would it flow through to like the when you're a funer what you're trying to do is buy Prestige by association with the people you fund you want to fund people who are seen as prestigious even if they don't actually produce progress you don't care so much about progress you care about the prestige you gain by funding them so if this person looked like they were prestigious but now these markets say they're not and people actually take these markets seriously as an indication [32:25] of who should be considered prestigious then they won't want to them as much I guess I'm just struggling yeah I'm struggling with that but I understand what you're saying in theory thanks so much for speaking to us today when did the markets resolve well they would resolve in the centuries later when you do the uh judgment but that's not actually a mechanical problem for prediction marks so basically any asset that accumulates value over time can be split into bets so you do not have to bet cash because cash you know depreciates you should in general be betting things that accumulate value like like stock index funds or bonds or something so we could basically have assets which are stock index funds except they're split [33:06] according to these bets about the ranking so these are things you should be willing to hold long term but of course no one person has to hold the long term for many long-term assets we have a sequence of short-term owners who buy it now and then sell it later to the next person and if the price goes up they're happy with the fact that they bought hold it for a long time the price went up so uh but it is true that a longer time it takes to settle a bet the harder it is to induce speculators to speculate because they have a longer opportunity cost but that's just something you can pay for by having a bigger subsidy look so people are definitely look so I have this work work on manipulation basically one way to [33:47] subsidize a market is for people to want to manipulate it and you can see how people would want to manipulate these markets I'm describing right like if you're a ambitious grad student and you want to be ranked higher you're going to be tempted to push up your own price right that's liquidity because other people know know you're going to be tempted to do that and they'd be happy to jump in on the other side and that becomes liquidity so the temptation to manipulate here would create liquidity all that requires is that people are taking the prices seriously and then then that liquidity would be produced and that creates the incentive for people to speculate what indicators will you look to do you think will will predict accuracy in the [34:29] future what what indicators are you focusing on uh so I mean the whole point of the market here is I wouldn't have to figure that out right we would just invite hedge funds or whoever to try to figure that out and then they would be making bets on it and we don't have to right like like if you say what are the indicators of a profitable company right now right well there's all these Specialists out there that's what they're trying to figure out all the time is which companies are should be have a higher stock price or lower stock price most of us don't know the answer to that maybe some of them think they do and they're wrong I don't know but the whole point is the system figures that out and no one of us needs to know the answer so I don't know what would in fact be the indicator that's that's the reason why I think this could work because if we all were really confident [35:09] what were the indicators there would be no point in doing this right we would just have the answers and we would know who is good but we don't know that's the that's the point don't we see this in the Capital Market involvement in drug development already where essentially private equities today betting on science early stage funding of science I'm an oncologist so oncology drugs this is a typical model of how private Equity that's science right we look at preclinical and early stage compounds invests in those companies and then those get acquired tra I I an economist I'm a big fan of capital markets and certainly there are areas of life where there's an overlap between [35:49] academic Prestige and capitalist Investments and in those areas capitalist investment prices will be correlated substantially with academic Prestige but most of Academia isn't like that and we'd like to pull that power into the rest of Academia and make it but even there you might say well sure that guy's going to make a drug that'll make millions of dollars but the future won't care about that see academic Prestige is all about dissing people who aren't you and saying sure that's you know Pat you on the head say sure that's nice for you but what we're doing is the really important fundamental stuff but if I get $50 million or $100 million haha that's prestigious anyway to some degree but in fact our world has a limited degree of prestige for money and [36:31] Academia collectively I think gains more Prestige than people with money do really in our world today uh I don't think that's just but I think that's the way it is you know in fact one of your main ways to get money is to get those academic people to give you a prestigious college degree right or a prestigious graduate degree that's one of the main most reliable ways people make a lot of money in our world right is to get the academics to give them a their you know metal of approval in the back yeah if you were trying to engage in Market manipulation what do you think the [Laughter] [37:13] best so the key point is that the response to your manipulation will be based on the expected amount of your manipulation so what you want to do is not be obvious about where you're expected to manipulate right if people all you know if you're the grad student and all of a sudden there's a lot of trading on your Market people are going to believe that's just you bidding the price up right so maybe if you're clever you'd bit on somebody else or something else that people don't expect to manipulate and then you can have more of an effect that's just the basic basic this is the look this is crime in general okay if if it's really obage which crime you're about to commit you're going to get caught the clever way to commit [37:55] crime is to do unexpected crimes right du yeah um I see on PO market like they kind of just like react to events um rather than like like the Shar is going up itself so um do you think it would be like a good way to prict if something happens could poly Market wait to predict if something mean like the events um like they kind of react to events they don't um they don't occur before people only bet on something in like large Capital before um the so happens actually you guys have heard of Insider training okay good you're in New York um basically you can look at prices just [38:38] before public announcements that firms make and typically the price on average does move before the announcement most public announcements actually are revealed via insider trading to the market so insider trading laws are just in Mass failing because in fact information is getting out on average before the announcement so um that's a direct demonstration of what you wanted right which is that in fact markets often predict things ahead of time uh why the preference for ranking people not ideas um because I I mean I'm happy to do all of it but people are our anchor [39:19] of prestige that is socially we're interested in the proced of ideas mainly because we're interested in the proced of the people associated with them is the IDE more long impactful um we're we're trying to do several things so we could have markets in the ideas and then you know you could have historians for example first I would expected historians would actually first try to say what were the ideas that mattered most and then try to trace that back to previous ideas that contributed then to find out who was associated with them and then they could rank those people that way uh but what we're trying to do is move the so basically a great many previous attempts to reform Academia have failed because they don't change academic Prestige they don't [40:00] change who is actually seen as prestigious and so the reason I'm focused on people there is to be as close as possible to moving that needle like to change who is actually seen as prestigious and for that I need the target to be the person so given that Target of the person what do you think about prediction markets on which late career scientists maybe midcareer scientists are going to get a Nobel Prize or mathematicians are going to get the fields metal because that would seem like a really good proxy in a way to start the market what so he's saying why not have markets and who will get The Nobel priz or Fields medal I don't have an objection to that um the question is [40:40] just whether that will push people better to have better incentives so does the question is when people finally assign the Nobel Prize is it because that something has eventually been revealed to actually be important and insightful or is it actually because that people have been fighting behind the scenes for decades and finally W somebody won the fight I don't really know let me State the the question again is that a interesting experiment to try to do to begin to get people to think and make forward-looking bets about what matters in science because it would be very I I approve of it certainly I I don't disapprove it would be a fine experiment to try uh I just tend to like figure out the solution and try to get there as fast [41:21] and direct as possible but that's certainly a fine intermediate step I don't at all disapprove uh a hand there no right here I ofs scary what would a list of assets that safest to bet how do I take profits and how do I gain money from this system so let me pick stepen Pinker at random Steven Pinker is a name you all you guys are all but the question is he overrated or underrated right that's the thing people often think about when they think about a particular name so if you're sitting around in a conversation and Steven picker comes up and you discuss him a bit you might get the sense he's bit he's a bit overrated [42:03] right especially you go look up his price and it says you know he's ranked this High we go it was good but not that good and now you sell right or you read a paper by somebody you go wow this is a really insightful paper let me go look up his ranking he's crazy low nobody knows about him by this is a good person to buy because the world hasn't appreciated just how insightful this person and contribution was that's the way you would speculate is by buy but and then they aren't the belts aren't settled for two centuries but the the market could move in the next five years as everybody else realized that Stephen Pier's overrated or that this other guy's underrated right so you don't have to [42:44] wait forever necessarily the question is how long it will take for other people to see what you see okay so basically you're betting on nerds either over or under they're prestigious academics you know I'm well I dare call them nerds no do you think there's like enough volume for this because um like this science is like a really Niche field and then like you're not going to get like a lot of people it's not Sports you're going to like that on experiment with that like individually would there be no volume to bet on about so we're talking maybe a million academics in the country on average making $100,000 a year right so we're talking hundred billion doll [43:27] Revenue industry here per year and these markets would be moving who matters how much so they could possibly get 1% of the volume of that value right so 1% of a 100 billion a year is a billion a year so I'm giving you a rough number and saying yeah that that could be enough so rather than having so the problem is uh you have a it's a marketplace right you have a buy and sell size right who's going to be betting against you because that' be very it liquid and I think the only way this could work is if you do like a casino where the house is the market maker and it has the OD think about say with grants apparently people [44:08] at the on average today are like applying for five grants for every Grant they get so there's this huge overhead in the evaluation process we're talking about a different evaluation process that would have a much lower overhead but yes any evaluation system is going to have some overhead but if it's if it matters if it is influential then it will have Revenue going through it because it's influential if people are using these prices to decide who to hire and who to publish and who to give grants to then people will want to manipulate them and that will create liquidity that produces the incentives to trade uh everything comes from are they influential would people actually look at these prices in deciding who they who they respected [44:50] that's the gold thing to get here if nobody cares of course then nobody will trade it won't matter right hey I want a question wouldn't this like overly favor the people that know how to like attract liquidity more than the ones that are actually like good academics because if I'm like very good at marketing but a bad academic I could still have look pretty right pretty much any way we could organize Academia will give advantages to people who are better at marketing themselves I mean I can't imagine any system that wouldn't have that feature so you know it's not you think we could get endowments for marketing endowments to Market make you think we get ends to Market what so in the history of science one of the main ways to fund science was always to to [45:30] not fund particular people or projects but just add resources and equipment like people often want to just pay for libraries at universities right they don't know who exactly is going to use the library how but they're just trying to add more resources to everything so I think that's those sort of people would be happy to just subsidize these markets they would just say I just want this whole thing to work better and I don't want to pick winners and losers I just want to support the whole thing and that would be a way to do it so if these if these markets really do um end up being considered you know um right and influential and everyone really cares about them then won't they eventually become regulated in some way and that [46:13] well most likely academics will want to take control of them that's what we've seen in the past right when outside institutions had a lot of influence where Academia academic said no you can't trust those Outsiders we need to be in control of that and maybe they would say oh the only people can bet in these markets are you know uh full professors at the top 30 universities and everybody else can't because that we can't trust anybody else's judgment they try to regulate and say only those people could trade me that would be the kind of move they might like to make so obviously the recommendations push against that but it might fail but like regulations against insider trading or Market manipulation these would also ruin price Discovery you can screw up a lot of things with over-regulating I'm absolutely I abely agree the whole point would be to try to avoid that but yeah [46:54] you you might not be able to yeah what are some ways to get people to see market prices as a source of prestige so that a fun might for instance increasingly weigh the market price when deciding to well for example the more prices we had about more other things so 20 20 years ago or so there was something called the Hollywood Stock Exchange which had betting markets in movies and acors which and if that was a play money market and then they were the people who would set it up were to make a real money market they' gone through all the regulatory approval to make that happen and then actually Hollywood insiders lobbied Congress to pass a new law to [47:35] make it illegal to have Futures markets in movies that's the only other the only other exception so far is onions sometime in the 50s the onion makers decided there should not be any Futures in engines and now there can't be any Futures markets in movies because Hollywood lobbied for that so apparently they were scared of what would happen if we had very legible market prices for actors and movies uh but basically the more people in the world that we have prices about that we see that in fact they're meaningful the more we would accept more sets of prices as meaningful so maybe Academia isn't the first place you might want to have this but this is this is the place you'd want to have it if you're trying to fix academic could you walk us through an [48:16] example of how um these future markets would work with say like Alzheimer's research where a bunch of like prestigious professors research was exposed as fraudulent like how would this facilitate that process of like getting to more credible science so as you may know in the stock market there are people who specialize in Short Selling and what they do is when they hear a rumor that some com companies a fraud they think oh there's a profit opportunity if I can go research that rumor and find out that in fact they're a fraud we can profit because we can be the first people to sell them short and we can profit by starting their price decline once everybody else hears their a fraud a similar thing had happened in [48:58] Academia I mean the whoever discovered the fraud in Alzheimer's Reems what did they get paid they got paid like they got to write an article or something in a newspaper like it wasn't a very big payout guys right if in fact there had been markets at all these people they could have shorted all those people and got a much bigger payout from their investigation so that would induce more investigation looking for ways in which stuff is Frau at the moment there's an awful lot of academic research that's just not very very good and just wrong and most people just don't have an incentive to go find it out like one of my first Publications was I noticed one of my professors in my department a paper so I looked at it and I said well that's a mistake and eventually said [49:39] okay yeah that's a mistake and so I got a correction but basically this is almost like a large fraction of papers just have mistakes and academic just but nobody's bothering to check for these things but if if there were more markets in you know your long-term reputation then that would make a bigger difference yeah um so unfortunate fact scientists sometimes die would these markets like continue based on their absolutely okay yep yep you can still buy and sell people are gone yeah so aren't the historians judging whether a particular scientist was influential also PR to the same biases that that Academia is right with [50:19] so couldn't they just kind of like rig the the system in a way or wouldn't they be incentivized to rig it we have many institutions in our world that rely on an independent evaluation done later by some people who are less invested in an area so auditing for example and accounting legal J juries uh and so the intuition there is that you get closer to an independent judgment uh if you move in certain ways away from the people directly involved so that's the intuition here is that you would do better so again if you just impaneled one jury to make the decision then you wouldn't necessarily know how bu but if you un panel several [51:01] independent ones then the difference between their evaluations is at least a a bound on how biased they could be because you know they they had separate evaluations but basically we're just trying to improve on as always we're always just trying to improve on the status quo yeah you know Perfection isn't an option so uh but but we could learn better how to make juries that were more independent and I think we should try to learn how to do that and this would encourage people to start doing more of that sorry so when you have the ranking of like the top thousand people how is the [51:43] pool of money distributed amongst the like people in that ranking so is it like yeah what is the power distrib or the distribution so what what you expect the historians to do is to give a literal ranking right and then the question is an asset pays off is it linear in rank is it exponential in rank is it logarithmic in rank that's about how big a difference you care about where so I would think well what you really care about is say actual intellectual progress no it's the door someone so I would want to roughly [52:24] estimate you know the function of rank versus progress yeah and make make the make the assets roughly linear in progress because what we what we care about is seeing differences in progress not really differences R exactly so I would want you could you would even you could even assign progress scores potentially but I think that's less reliable than having them sign rank scores that is again if we're looking about bi concern about bias or games they could play Just forcing them to rank order people with be a discipline and distribution right but basically you could do other research that would try to guess what's the rough relationship between Rank and contribution and then you use that to to set the rough scale that would be my recommendation yes I'm [53:05] curious how you'd um get markets that Fai in the decisions of human actors subjectiveness I I think there's a good knowledging with the secur overnight financing rate features that we see with like fed policy because it doesn't matter if the FED makes a mistake or not it's like ultimately the contract's going to settle to whatever they decide what here this is going to be less to do with like I can seeet matal like this to pricing and what the human actori panel matur what they decide but how does I I agree that that would maybe potentially become mat would ultimately jeopardize the origal point of project perform Academia rather just creating an opportunity for corruption the Jes so an [53:46] analogy might be museums and paintings or sculptures so when museums choose to invest or Buy in a painting for sculpture that's basically represents a weighted average of expected future Prestige for that person right uh they're trying to guess not just who people right now want to see in the museum but who people in a generation might want to see in the museum so that's at least showing that there are worlds where we have these Market evaluations and long-term judgment right similarly like for historic buildings people invest in by historic buildings right not just based on what they can use them for now but based on a judgment of into the future how much will people [54:27] respect that particular historical building and again those are examples where we have market prices that represent substantially a long-term estimate event and of course we actually have like 40-year bonds somewhere so that's definitely estimating like the chance of paying off in 40 years in the inflation right so we we do have Assets in our world that represent long-term value but in those cases the I is private like the the work itself like the sculpture is a privately owned thing as opposed to right but it's based on this public reputation right the reason you want a b go is because everybody says van go is great you're not buying it so much for your enjoyment of B right you're speculating on what everybody else will [55:08] say about B maybe then it would be better to instead of having raw ranking like have a tier list something a little bit more qualita sure that's another possibility and that might be easier on the judges but I still might say okay you know put 1% of them in the top tier put 5% of them in the next I give percentages that have to go in the tiers and that's another way of ranking right but it's a little easier for them I might that might be fine then they could be betting on which tier that we wa so we know the government's willing to ban Futures markets in Hollywood right Hollywood lobbies yeah lobes right think they'll be to Futures Market in Academia once it takes off [55:48] they know you're trying to rest hour might but that would just tell us we're on to something right you know if I were trying to reform Hollywood I'd say oh they were on to something you got Hollywood to like pass a bill to forbid your thing your thing might have been a pretty good idea long for Hollywood or for fut I mean that's up to you guys you're you're the Next Generation I'm 65 years old so I've got another maybe 20 years or so and you know this probably won't happen in that time so it'll be up to you guys what about for Hollywood uh there might be something sooner you [56:29] know you could imagine doing something on crypto for Hollywood now I guess I'm not sure why nobody's doing that really that was it really success in prediction Market you yeah I had a question I'm curious if you see a problem with this if I'm an individual actor trying to profit in the very near term in one of these prediction markets then more so than betting on the academics long-term importance historically I'll be looking to bet on how everybody else will see this academic in one week right and do you see that as not optimizing for truth or not finding like a closer uh robustness of the scientist's work more so how good they are at convincing [57:09] the public that they are a robust scientist so I'll ask you to consider two counterfactuals you're imagining doing this strategy one counterfactual is where you do the other better strategy of focusing in the long run the other counteract you do nothing in these Mars you just leave them completely an right I'll say your strategy is better than the doing nothing so basically our standard math models of financial markets have two kinds of Traders there's informed Traders and uninformed Traders I.E what I'll call wolves and sheep the simple model is the wolves go over the Sheep part so when there are more uninformed model traders in an area then there are more informed Traders there who are coming to profit off of the uninformed ones so you are then [57:51] trying to be an uninformed Trader there right or less informed and then that will attract other people to you know counter you so there's plenty of room in financial markets where people specialize in lots of different things I don't want to tell you you shouldn't specialize in any particular thing I would just say if there's too many of one kind and too many of another then Market forces will discipline I'm I'm thinking kind of like the uh speculative bubbles that happen with meme coins or like meme stocks and how step picker could be a speculative bubble right for a little while but then the discipline is if you think it's a bubble then you should sell it short and profit in the long run that's just how financial markets work they're not perfect just want to make sure it's kind of mechanically understanding the concept [58:31] so you're saying like you have a panel of you know History of Science folks and then like every year or whatever they come up with like a one to a thousand ranking of like impact and then the markets are you know kind of um pricing and like resolving around that so I can enter into you know a v which is like who's going to be number one or whether this person's going to rise what's the so basically every academic will have an equal chance that sometime they will be ranked maybe most of them will never be ranked but you're betting on if they're ranked what will the rank be so then in the future what we do is we pick random areas and we rank everybody in that area [59:13] and then we pay off the assets that we're betting on those ranks and then other people their assets are you know called off because they never were ranked so we wouldn't actually have to rank everyone we just have to create a situation where everybody has an equal chance to be what happens when you get to go to do the thing that you were really excited to about where you thought that all the people that were there were trying to actually do the one thing and then it turns out they were not actually do the that is as an as a specula as as an um I guess like the thing is like yeah gu it's just like I don't want to be a Speculator I guess and it's just like but you'd be an evaluator basically that so in this world a spectator is an [59:54] evaluator as you are going in and saying what do I believe here they they've been making a lot of claims like I'm going to go in and try to judge what's actually valuable here and that's a role we're creating an incentive for people to take on they aren't taking on now so much and that's valuable you don't have to take on that role if you don't want to again you know the world has room for lots of roles you can find your role in that world you don't have to pick the proper algorithm to do that necessarily but the world will just fill different roles and if you find a role that's underfilled then that'll attract you more to fill it because there'll be more R rewards there but it's just not us then don't do it yes sorry just like following through so then you're saying so like in five years it's this you know Stephen Pinker is on the list or not on the list and then 150 [1:00:37] years but what we're trying to influence is like today who you know Cambridge Oxford Harvard decides to like hire or whatever so we're making bets which are you know kind of revealed over some number of years or decades so how does that and then you 70% believe that they'll be on the last 30 don't but we'll find out in 150 like how does that put pressure on like Harvard to change their decision making today if we had a market price in Vos and the Vango price goes down that would tell everybody oh looks like speculators think Vango is a flash in the pan not going to be a longterm thing and they will then be less interested in Van go that's how it works or but I'm [1:01:18] Harvard I'm like well you're just idiots and then maybe after like some decades I I guess you're saying this is like a long game decad like hard you've been for the whole point here is AC the lever I have on Academia the only one I could find is their claim that their people are the people that will in fact be seen as the best people in the long term and that's the lever I'm trying to use but it's a long-term lever and I got to use it as it is as a long-term lever yeah yeah what metric would you value academics by what metric would you value academics well for this mechanism I don't have to choose an absolute value I'm just doing these rankings remember so person WR researchers you I mean if I were a [1:02:01] Speculator how would I go about it well I would do it as a side effect of just when I'm reading and doing other things I come across somebody and I think well that's a bozo and I look up the ranking and I go that's too high and I short them or I read something I go wow that's pretty impressive I look them up and I say that's low price that's a good investment so I would and the fact that seems to me if there are people who are studying these things anyway they just have a lower cost of getting information about this you sort of having a whole job of studying it just because you're paid to that would be more expensive to get you to do that so if there are people just doing it anyway then they're the lowcost provider of this information right say for sports there are people who bet in sports sports right the lowcost provider [1:02:42] of information about sports is people who are already into the sport already watching the sport already thinking about it they generate opinions about who's going to win which games and they bet on that it would be silly for somebody else to hire someone who had no interest in sports go study that because they're more expensive right it's harder to get them to pay attention if people are already studying it for some other reason then they're a cheaper source of information so why companies are getting more prestige in terms of attracting scientists and producing good quality research uh and dealing with problems that have problem like with the pandemic for example no price LA this year of chist [1:03:25] came from industry my last year medicine PR was also from industry so how do you see the future of Academia why being in competition with the research that's producing in outside of Academia I wouldn't call industry outside of Academia I would I would Define Academia as the circle of people who rate each other right it's the mutual admiration Club some of those people can be in industry and they often are some of the people are in universi some are in think tanks what matters is that they maintain this control by being the ones who decide who's good that's who I would say [1:04:07] is the Academia but they can be an industry that that's allowed they often are but but the key thing is how do we break this Mutual admiration Society of them being the only people allowed to decide who's good for their stuff and they get lots of resources channeled through them because they're the people who say they're good that's the question how do we change that would you say like H index or citations would still be useful metric in future sure but I mean that's a low bar I a useful metric sure but how useful would be the key question so speaking of actual impact of those prediction markets on the Academia [1:04:47] world like do you think that we will need like a totally different set of infrastructures like Publishers like reporters because if not then if the the control will still be under those people who are already considered prestigious in in the Academia World which they will fight fight against you know all those prediction market results like in a say because it's like conflict to their interest so I've in a number of different areas happy to tell you about other areas tried to invent reform institutions to reform things that seem broken to me and a big question is how broken are they and what I like about a reform good reform institution is you don't have to know the answer to that [1:05:28] you create the reform incentives and then it will tell you how broken things are if things are just a little bit broken then they won't change things very much because the good incentives they will that they respond to will in fact make them not do that much but if things are really broken then they will do a lot they will make big changes because in fact things are really bad and so when you take bad incentives and you give better incentives one of the things you'll find out is just how bad things work and I don't think we know that now we don't know how bad things are and we won't know until we actually get better incentives to and see how people respond to them lder yeah um how do we know that academics are like truthfully reporting [1:06:09] that preference for like 100y year time frame Prestige as opposed to optimizing for like prestigious industry St like stature right now like things like that like are we sure that's the real metric that I mean your model of Academia at the moment could very well have most academics not caring very much about the long run but my claim is they won't admit that their Public Presentation is that they are the people who will be seen in the long run that's what justifies their resources and Prestige that's their claim but of course if they can get away with just making that claim and then ignoring it and just focusing on short-term payoff they probably do [1:06:50] that would be my prediction right but we're going to go after that claim so how does the market deal with like tectonic shifts in what we consider Prestige and so like a historical event like move fast well like rosn Franklin right like for a long time she wasn't considered prestigious and all of a sudden she's prestigious because we've had lots of social events right so what's to say if all of a sudden higher ranking academics all of a sudden you know we don't care about sexual harassment then we care about sexual harassment how do how does that market take into account like that I mean we have a lot of experience with financial markets right and they are they can be very fast and response sometimes right [1:07:32] when the underlying things are changing fast but this is prestige so like right academic researchers who had bad behavior in the past but were academically considered prestigious were prestigious but now they're not prestigious because of their bad behavior well we we're looking around we're wondering whether other people care about their bad behavior and in a world where they don't care we don't care either but we might find ourselves in a world where we think other people do care and all of a sudden that'll change what we think other people will do so that's these are just very common things that happen in markets an interesting thing to notice is quite often in markets declines in reputation are often much more sudden and large than Rises I would think that would probably be to an Academia you know you have to spend a long time [1:08:13] slowly make building up your reputation but you can lose it in a day with if you do the wrong you know with the wrong sort of thing comes out uh well this is more of like a comment on the previous question like one way you could solve this is by like when you're talk when you're defining Prestige it might be like in terms of how influential someone's ideas are not how like influential not how like good that person is specifically so even if you know due to social factors that person falls out of favor they might still be considered prestigious by that definition right but for example I saw a recent thing that claimed many of the recent AI advances were were just plagiarizing stuff from 20 years ago somewhere else okay I don't know if [1:08:54] that's true care but you can imagine if it's about who really came up with stuff and should have been listened to and we find this person everybody's been listening to and it was good stuff but they got it somewhere else they didn't admit then you might pretty suddenly update your opinions about them there right that's about how much what they did actually mattered but that would be less about their you know politics or personal Behavior than about again again we have to pick a metric so I'm presuming we would want to pick a metric where historians aren't looking so much your personal behavior when they're judging your right act but that's something for historians to decide I guess so are the rankings static like Howa would they change well the rankings [1:09:35] wouldn't happen until 200 years right so and there's only a chance of it happen so in 200 years some group will get together and they might include you in their ranking and at that point that your asset will pay off it but up until that point there'll be this market price that fluctuates every hour for 200 years until you finally get the asset pay on all that so someone mentioned like dead people old researchers still you could still until they're 200 years later evaluation you can still trade them trading cards right so it sounded like you were saying that the majority [1:10:16] people so that depends on our budget for doing so I'm trying to make a robust system that if we end up not having a very big budget for this we can still handle it how many people what what percentage do you need do you think like how would we find that out and given that that's something that's going to be factored into people deciding whether or not what we'd mainly care is the thickness of the markets today and that we could see that immediately today so if you know people you could have a market estimate in the probability that on average people would be selected and if that number was low then that might make a very thin market now we could see that in the thinness of the market now and then we would know I guess you know we need to up this probability in order to get more trading now so that would be something you'd see [1:10:57] immediately you don't have to wait to judge that liage you would see are we getting enough trading now in these things basically what we'd want is that you could I mean presumably the manipulation is the thing like if nobody cares about these markets nobody will even bother to manipulate them right but if I can double my reputation for $10 it's going to be pretty tempting as an individual researcher to do that so if people are taking it seriously I won't be able to do that it'll cost a lot more than $10 to double it might cost 10,000 at which point we're seeing a thick Market you see that now is thick enough to get people paying attention seed well the first project [1:11:38] would be these is to prove that we could take stuff from two centuries ago have several different groups try to rank them and come up with consistent rankings that would be the first thing to show that we have a way we we can give instructions to these future evaluators that that they could follow that we believe would be you know coherent that would be the first project and then the second project would be to pick a particular area to start to some cryptography or something and you then you fund markets for the people in that area and try to show that you can get those thick enough and then you can move it to New areas that would be the order I would recommend how [1:12:19] that the the historian studies I'm pretty sure you could do nicely 10 million maybe you can do nicely for 1 million I mean that's you know historians aren't that expensive actually there's there're plentiful Supply they aren't that well occupied already so they they could be bought cheap it's not that to insult them I mean that means like hey let's use the more it seems like you're also making a bet on whether or not these institutions will persist for years so why should I believe that they'll exist longer than Nations um if you create a financial institution whose purpose is to last 200 [1:13:01] years it's much more possible to make it last 200 years so for example Benjamin Franklin us you know founder he actually funded several uh pots of money by two I guess it was Philadelphia and Boston Each of which was designed to accumulate for a century in assets and then over the next century spent out subsidizing apprenticeships and those both happened cuz he paid for them to happen um actually the world could have a lot more long-term financial institutions but we have prevented them by law so an interesting counterfactual [1:13:42] world to imagine is uh imagine you were able to in your will fund a foundation that could just take its money and reinvest it every year and didn't have to spend it on anything those foundations would in fact grow and come to the economy they would lower interest rates increase Capital rates and we purposely prevented that LE Court ruling purposely prevented that because they thought that is the Dead Hand of the past ruling us now we don't like that so we are going to ban those but we could actually now have more long-term financial institutions than we do we prohibited it by law how do you you're talking a lot in terms of long-term what would an opposite thing because people like [1:14:24] shortterm returns right how would you recraft this to be more shortterm to for the purpose of incentivizing people to actually use it maybe from a practical perspective so every research project or experiment or whatever has some people involved and then it will have some ending and we could bet on the ratings of people at the end of that experiment or whatever conditional how the experiment turns off right that would be a way that you could directly bet on short terms right because basically some people will bet on how the experiment will turn out and other people will bet on condition in the experiment turning out very much how much will that affect those people's reputations and in general um there are ways to make comod tutorial markets that we don't have at [1:15:04] the moment but you could have basically large Bay Nets of com of representing probability distributions over many things and then people would specialize in different parts of the bay Nets representing different beliefs they have and expressing it I'm just describing one example of that here so you know these eventual evaluations are only one end point of a causal path that leads to them and people could be better on all sorts of parts of the causal path that gets there and then the liquidity that subsidizes the final end points would add liquidity to all the paths that lead to it in terms of letting you go those what are you paying historians to do like today not the teacher but today we pick some area of the past cryptography maybe and we would assign a [1:15:47] group of people we call them historians they don't actually have to be professional historians but they would go to that area of the past find all the people there read their stuff look very careful discuss it about and then rank them in terms of who should have been listening to that's their job they come or the ranking or discreet levels maybe but still that's another way of lower resolution ranking and that's their job is to come up with that ranking but how does that play into the prodiction markets going forward we're trying to show that we can have the markets reliably payoff so I will fund five different groups right I'll fund five different groups to look at the same historical period not talk to each other come up with their own separate rankings then we'll see how correlated are these [1:16:27] rankings if they're pretty correlated that tells you look we could do that for the future too we'll have some noise but we'll do a pretty good job of ranking people yeah so like one idea that this might Inspire would be like if uh you had a journal which um had its editors or had its reviewers bet on Whose studies would replicate before publishing right and like yeah I told you I already we already failed to get them to do that oh really that was my story initially right so I said I was part of a project where we did in fact show that we could predict which papers we replicate and we wanted to get journals to say they would use that as part of the review and they said no oh wow they do not want to not [1:17:07] publish things that w't replicate but see that's kind of weird because like you would think you know especially in fields like AI where there's so much industry incentive to have a replicable papers that it would actually boost their Prestige you might think so but we failed to get any journals to sign on to this uh you start your own journal or you could but again you're they're making an estimate of market demand I don't know that they're wrong about their estimate of market demand my guess is they're right so I'm not going to bet against them so you you came up with a working prediction market for replication of p i was part of a group of people I but we had we did prediction Market showing that we could predict which papers would replicate we could not get journals to say they would use that as part of the review process and [1:17:48] what did you learn about marketing prediction markets that affects how we would do this with the need customers who care about the truth if you can't find a customer who wants the truth that you're estimating then you don't have a product right and it is much harder to find customers who want the truth than you would have thought most people claim they want the truth but they're lying it seems it seems like that for a specific paper with industry there'd be a lot more people like I need to know I'd be willing to pay million dollars to know if this is true with Prestige it seems much harder yeah that's kind of what I was getting at as well you might think so that's that's the impression they put on but they're lying I think maybe targeting for Academia because Academia wants more click baity [1:18:28] stuff that they can sell to the Future whereas industry wants things that will save them money and be useful most large research Labs like those at Microsoft or Google or Xerox in the past or B labs in the past the main reason they're funding those labs is for the prestige those labs give to the companies not for the particular research results they produce so if they produce stuff that doesn't replicate the gives them Prestige they're happy for that to happen so Harvard has a lot of prestige so the top talent goes to Harvard so is Harvard uh prestigious because the top town is there or is it prestigious line [1:19:09] itself and how will a prediction Market change this um type of thing Harvard says that their people are the best the rest of us dare not disagree but the top talent will go there because everybody well the top talent they will choose whoever they're going to label the best they don't necessarily have to choose the Top If the top talent is too cogy and difficult for them they very well will reject it and choose some more Cooperative lesser talent to to label top again they have the power to say who's top and so they will use that power at their convenience right but if we wonder if they're in fact are they the top that's the point of these prediction markets is [1:19:50] they could highlight that they're in fact not picking the top when previously we didn't know when we weren't so sure we couldn't disagree with the public but now we could because the prediction marks are saying no those people are not talking that yes as the idea has been explained I feel like the market is likely to be somewhat liquid IL liquid IL liquid however if you made a small change I feel the liquidity could explode which is explain why so let's suppose that rather than being researcher focused you make it issue focused I think some for example could have a project focused like paper focused or experiment focused or [1:20:31] something carbon capture or climate change so by making it issue focused all of a sudden you motivate really big corporate companies like British Petroleum they might want to hedge their climate exposure their carbon capture so they have a natural Inc that was my initial Vision 30 years ago 35 years ago actually and it's most people's initial Vision when they first get in prediction market and it takes us a while to learn that that doesn't work the problem is that it only works if those markets are thick and have lots [1:21:12] of activity but if the prestigious academics go that's useless and worthless and dismiss it and don't consider it than nobody else does either that is most academic discussion get attention because of the prestige associated with it and the markets can only have attention if they also have Prestige associated with them and so somehow they need enough liquidity for that to happen and the academics will not lend their Prestige to they see that as a threat and they reject it how specific and how rigid should instructions to Future evaluators be and do you think in 100 years AI will be better and more efficient with [1:21:52] implementing these the reason to do these initial experiments is exactly to see how robust it looks like the results are to different methods and different instructions right obviously the more over time you did more research like that the better you could learn what better instructions you and now the question might be initially do we say here's our initial instruction but we expect we expect the instructions to improve according to these people introducing new instructions will people be okay with that well they want you to specify the instruction at the beginning that that's L's thing I feel like I've taken a time here okay somebody in charge of any last questions okay well um yeah two seconds go yeah if I think about like the advances that are like shaping the [1:22:34] future today you know Gene sequencing criser aom Vehicles AI stuff like they're pretty much all from like the most prestigious institutions so like what's your right like the your argu those are the prestigious things which come from that's that's why they're prestigious well they're prestigious cuz they're the things that are changing you know the like according prestigious people yes how do you know which of all the things are happening are actually changing the world well you listen to the prestigious people they tell you the things to look at and those are the things they're telling you so yes those are the things you hear about there's other things happening by the way yeah exactly so like what like what would you like what's your example I mean I'd have to go search for it right I mean that that's I mean I could tell you in my areas I can tell you there's nobody at [1:23:14] Harvard doing what I'm doing talking to you about right now amazing okay so what's your contact how can people get in touch tter I'm easy to find and search on my name I'm on Twitter email okay well I'm not hard to find and so we got you this Mi science grade again hat okay um wait wait will New York people read the words before they mug me and then a e NYC um Beaker mug for you as Weller mug yeah so you can put your coffee and measure milliliters it has in there right um You have a pip pet stir um no but I'll send it to you yeah [1:23:57] um so let's all thank Robin I think that was the longest e NYC so we we stream after it's know it's a little hot in here and thank you guys all for coming as well um we'll see you all tomorrow at D NYC Tad um and yeah enjoy continue discuss and get with us get talk to each other