Stories from a Top Forecaster: Exploits
Part 3 in the 'How I Forecast' series from Top Forecaster, Joey Shurtleff
Forecast is a new community for discussing the future, organized by topics that matter to people. Whether about news, politics, social issues, sports or pop culture, Forecasters work together to share their knowledge about what's going on today to build a better understanding of what will happen tomorrow. Download the app to join the community.
In this 3 post guest series, we welcome Joey Shurtleff, Forecast's all-time top Forecaster. In addition to being a top Forecaster with almost 2 million points, 400 forecasts and 900,000 points in reason supports to his name, Joey is also a serial entrepreneur and world traveler. In part 1, Joey discussed his analysis strategy, and last time, he detailed how keeping up with world news and other data sources has influenced his forecasting. In this post, he’ll discuss exploits.
In the third and final post in this series, I’ll focus on inefficiencies and idiosyncrasies in the Forecast app that I’ve been able to capitalize on, along with some product improvements that could mitigate them. These are divided into five forecast types: squatting, errors, flipping, sure things, grabs, and sure things.
Squatting
My most lucrative forecast type, beating any of the forecast types described in my first two posts, was squatting. This is when I notice a new market has just opened and I “squat” on what I believe to be a good position, typically with the intent of selling this position at or near the “true” price. To spot these opportunities, I refresh the app frequently during the hours in which questions are typically released, which is roughly during business hours on the US west coast. I gained over 180k pts from this, representing over 22% of my total gains, with high-conviction squatting representing over 95% of these gains.
Some squatting is easier than others. For example, this question was my most lucrative squatting, with over 8,500 pts gained from this position.
When the question was introduced, Filecoin futures were trading very comfortably above $12, and therefore the position “$12.01 or higher” was almost a sure thing. I purchased 200 shares for 11,588 pts, and realized full profit when the question was settled less than 36 hours later. Other lucrative squatting includes “What percentage of the national popular vote will Donald Trump receive in the 2020 US presidential election?” and “When will shares of Palantir Technologies Inc. be publicly listed on any exchange?” (the listing date was announced hours before the question went live).
Occasionally squatting has backfired on me. The biggest example of this was on this question:
I grew up in the Bay Area so I thought I knew the way this market would move, but it turns out I was wrong and I exited my position with a loss of 1,247 pts.
Product improvement: These opportunities have gotten harder to come by lately now that there are many more forecasters on the app. Nevertheless, I believe that the Forecast product team should work to reduce the magnitude of these opportunities going forward. There are many ways this could be done; the simplest way may be to limit the number of bids that a forecaster can make on a market (or on a given position in a market) in the first N hours after the market is live. This change would ensure the gains from lucrative squatting would be shared amongst the fastest several forecasters instead of mostly going to the fastest forecaster with enough points available to fully capitalize (who has often been me).
Errors
My fifth most lucrative forecast type was “errors”, when I capitalize on unusual and possibly unintentional or uninformed forecaster behavior. The two most common sources of error I’ve identified are forecasters selling irrationally large quantities of shares at once and forecasters misunderstanding the settlement details of a market. I gained over 70k pts from this, representing over 8% of my total gains. This accounting understates my true gains from errors because I’ve based the accounting on why I bought, not why I sold; gains earned from shares bought at fair value and sold at inflated valuations due to errors aren’t covered in this analysis.
One of my largest gains on errors was in this market:
On election day, someone accidentally sold 722 shares of Trump. When a forecaster buys or sells shares, the market prices move accordingly; buying or selling 10 shares will move the market price 4 or 5 pts when the odds are close to 50/50, and less when closer to the extremes. This large sale caused the market price to move to its lowest possible value of 0.1 pts per share. I noticed the error about a minute after it occurred and I scooped up 300 shares at an average cost of 1.36 pts per share. I sold all 300 shares within 20 minutes of acquiring them and gained over 8,600 pts in the process.
Another example is this question
The settlement details specify that there must be at least 50 Democrats based on ballot-listed preference, which means that the two Independent senators that caucus with the Democrats don’t count. (I wrote this question but I didn’t foresee this complexity and the ballot-listed preference part was added by the Forecast team.) Many forecasters bought the Yes side of this market seemingly missing this important detail, even after I, and others, highlighted it in our “reasons”. Eventually the market reached a more rational equilibrium and I gained over 9,500 pts once I finished selling my position. This market remains misunderstood, and I’m still buying shares of this question today.
Product improvement: One of the main reasons why I catch large market dislocations (many of which are due to high-conviction forecasters rather than forecaster error) more quickly than other forecasters is because I follow almost 2000 other forecasters, which means these dislocations show up in my main feed. To enable other forecasters to catch these dislocations without following so many people, I think the app should enable forecasters to set price alerts and get notified when a price rises above or drops below targets that the forecaster can set.
Recent product and user base changes have had a significant impact on this forecast type. Over the last month or so, I’ve noticed fewer large sellofs, perhaps because the number of shares to sell now defaults to no more than 10. On other other hand, I’ve noticed many more large purchases; this may be due to the new feature which makes it easier to buy shares in bulk, as well as a recent rapid increase in new forecasters, who may be less aware of how large buys impact prices.
The Forecast team has worked hard to show forecasters how purchases impact prices but my experience suggests that there is still ample room for improvement. The title of the buying screen says “I think there is at least a X% chance the answer is: Y”; this is often misleading, particularly when a forecaster is buying or selling in bulk. For example, let’s suppose a forecaster wants to buy a position with a current list price of 35 pts. The forecaster opens the buying view and sees the title “I think there is at least a 35% chance the answer is: Y”. if the forecaster buys 60 shares at a listed price of 35pts, the first incremental share is purchased for 35 pts but the last incremental share is purchased at a price of ~64 pts. Thus, this purchase really implies that the forecaster believes that their forecast has at least a 64% chance, not the 35% chance implied in the title.
Also, there are a significant number of forecasters who, from my perspective, appear to base their forecasts on ideology and/or wishful thinking rather than engaging in any sort of well-reasoned forecasting. These forecasters often make the largest and least-defensible buys, and taking the other side of these positions is typically highly lucrative. This is particularly true in markets pertaining to politics, but also applies to other polarizing topics like Bitcoin and Tesla / Elon Musk. The Forecast team could consider only enabling large buys for forecasters that have been on the app for some period (say a week) and/or for those who have earned certain achievements.
The problem of misunderstood markets is a challenging one because there will inevitably be some questions that are released with settlement ambiguities or oddities that escape detection. I think there should be a new view in which the final text of new questions is posted 24 hours before launch, giving forecasters like myself an opportunity to catch ambiguities and oddities before they go live. Also, I’d like for Forecast admins to consider adding a prominent notice to questions that have been flagged by the community as potentially misleading.
Flipping
My sixth most lucrative forecast type was “flipping”, which is when I buy shares with the express purpose of selling them at a higher price (though I still buy shares at a price which I believe to be fundamentally attractive). I gained over 60k pts from this, representing over 7% of my total gains.
The majority of my gains from flipping came from one market: “Who will be elected president in the 2020 US election?” I gained around 100k pts from this market overall, and roughly half of that came from flipping shares (most of the rest came from two large forecaster errors). I made a point to follow forecasters who would buy heavily on one side of the market (usually pro-Trump but there were some heavy pro-Biden buyers too). Whenever I noticed that there was a significant price dislocation in my feed (usually favoring Trump), I would buy the Biden side until I brought the price to a reasonable level (which I considered to be 70 to 75pts favoring Biden). Then, I would wait for the Biden buyers to inevitably start buying and I would sell while keeping the price in the 70s. This strategy enabled me to generate my largest gain on any single market while not holding a single share when election results started coming in.
Many forecasters would have held onto shares acquired at a good price, instead of selling them once the price increased. My strategy is to always prefer to sell at a price that reflects, or nearly reflects, the true odds rather than holding and gambling on the final outcome. This strategy also depresses the market price of my position, which means that it may only take a purchase of 10 to 20 shares in disagreement for the market to move back to a lucrative price for buying. Because many forecasters thought the odds of Biden’s victory were higher than I did, I was more than happy to sell my Biden holdings to them, and wait until pro-Trump buyers swung the market in the other direction to buy again.
I generated more modest gains flipping shares in the market: “Will the Dow Jones close higher on November 4, 2020 than on November 3, 2020?”. I noticed that the market tended to revert towards 50/50, so I bought shares whenever one side dipped below 40 pts and sold them as they approached 50 pts again, generating over 2,000 pts in the process.
Product improvement: As mentioned in the Errors section, I think the app should enable forecasters to set price alerts and get a push notification when a price rises above or drops below targets that the forecaster can set.
Grabs
My seventh most lucrative forecast type was “grabs”. I use this label to describe two times when it was particularly easy to grab free points: during the Emmys and on the day after the election. “Grabs” on these two days have generated over 55,000 pts in gains for me, representing over 7% of my total gains.
During the Emmys, many of the markets were closed prior to the event but 6 markets were not. I watched the Emmys and bought shares in the winners once they were announced, and thereby gained over 10,000 pts.
On the day after the election, the Forecast team made the decision to unfreeze most election markets which had been frozen during the election. Many of the races referenced by these questions had already been resolved, and some of them were significant upsets. I noticed this before almost everyone else and rushed to buy shares of the winners, and thereby gained over 45,000 pts.
Product improvement: Markets shouldn’t be unfrozen when the outcome has already been determined. I’ll talk more about freezing and unfreezing markets in the next section.
Sure Things
My eighth most lucrative forecast type was “sure things” or “almost sure things”. I gained over 30,000 pts from this, representing over 4% of my total gains.
Naively, one would expect bidding on a “sure thing” to offer very little value in an efficient market. This is often true, but there have been a number of instances where there was ample value to be had.
The most lucrative “sure things” have been questions that were seemingly overlooked by other forecasters because they referenced an obscure topic and/or because they were old questions that took lots of scrolls to reach in the navigation. One way I find opportunities like these is to sort all questions by “closing soon”. I gained over 2,200 pts from “Will the Brandon Act become law before November 3, 2020?” and over 1,500 pts from “Will Alcor Life Extension Foundation have more than 2,000 members by the end of September?”
Some value can also be found in less obscure topics. Forecasters have a limited number of points to “spend”, and “spending” 99 pts to win 100 pts a month later will likely offer a worse return over time than other higher-risk bids. Because of my strong performance in the other forecast types, I have often had more points than I can allocate well to other forecast types. I’ve parked these points in “sure things” because they act like a savings account, offering a safe low return on investment. For example, I accumulated large numbers of shares in questions like “Will a presidential debate take place on 10/7?” (this was the day of the VP debate), on which I gained over 1,400 pts, and “Will Joe Biden drop out of the presidential election?” (this was exceptionally unlikely when I started buying shares in early October), on which I gained 700 pts.
Product improvement: Although I’ve generated a non-trivial amount of gains from “sure things”, the relatively small percentage of my total gains this category represents (and even slimmer percentage from “obvious” sure things) and the slim per-question gain on most of these forecasts suggests that this isn’t something that has been a major contributor to my total gains. For this reason and others, I believe that the Forecast team should err on the side of leaving markets open too long rather than closing them prematurely.
This was particularly true during the election, when many forecasters initially enjoyed trading on the news as it was becoming available, but had to stop prematurely once the markets were closed. To make matters worse, the markets were reopened the next day when most of them were already “done deals”, which created the highly lucrative opportunity to “grab” points I detailed in the Grab section above.
Allowing forecasters with lots of available points to buy shares in sure things also enables forecasters with fewer points to sell their “effectively resolved” positions at 95+ cents on the dollar instead of waiting weeks or months for settlement. I noticed many forecasters doing this on the days following the election. For whatever reason, the Forecast team took longer than necessary to resolve some election questions (many states were called by AP on election day but the relevant markets stayed open for a week after the election), but forecasters had the option to cash out at ~97 cents on the dollar because I used some of my points to buy these sure things at “almost sure thing” prices since I couldn’t find anything more lucrative to do with the points.
In Conclusion
One thing I hoped to learn from this analysis was what portion of my gains have been generated by true value-add forecasting versus capitalizing on inefficiencies that many other forecasters would have identified. I would prefer that the performance of myself and others be measured based predominantly on value-add forecasting and, in the analysis above, I think the split was roughly 50/50. I hope the Forecast team will prioritize the product improvements to make squatting, errors, flipping, and grabs much less impactful on forecaster performance going forward.
Otherwise, I hope that this series has been insightful for current forecasters and has piqued the interest of others. I hope that the Forecast team continues to grow engagement and improve the product, and I hope that the analyses above help forecasters refine their approach. Happy forecasting!