In general it's better to be safe than sorry, so spending a bit more on predictions before launching a product is a good thing in my book. Asking employees is a great move. At first I simply assumed their predictions would be worthless as they are all biased. But then it came to me - they are biased with, so to say, knowledge! They know the product (since they made it ..) and know better than anyone else it's faults. That's the reason for the disturbingly high statistics. The only reason I can see not to predict is when you're occupying a niche market and have your faithful customers.
Tad Milbourn
· 4 months ago
A very interesting post and fascinating data. The question running through my head though is the extent or limit to which prediction markets can be used. At first glance, it seems that those participating in the prediction market need to be a good proxy for the customer the company running the market hopes to target.
In the Electronic Arts example, their front-line workers are often gamers (EA's target customer), so they serve as a good proxy. The often cited Best Buy "Blue Shirt Nation" example also holds to this. Best Buy employees are reasonable proxies for the electronics enthusiasts that shop at Best Buy stores.
Does this resonate with the data you're seeing? Do participants need to be proxies or "informed enough" to make valid decisions in these markets? Is the crowd more accurate even when making predictions outsides its collective expertise?
Arik Johnson
· 4 months ago
Since reading the Times story I've been wondering: Why anonymous? If it is to maximize participative liquidity in any given market where insider information would prove salient, that's answer enough; however...
With government regulation looking like it'll only increase going forward (at least in the U.S.) it strikes me that corporations seeking to aggregate knowledge in the enterprise using PMs as a method will need to know "who knew what and when" more than ever before.
The Times piece and subsequent comments loiter over this point of anonymity as a primary asset and differentiator (which I acknowledge by the way), but the upside escapes me if it's impossible to answer the who-knew-when question.
Other than removing disincentives to participate where insider knowledge to game the market might prove compelling for the insider to remain anonymous, the reason regulators will seek to uncover them is to present a paper trail to call out these insiders in the event of impropriety. As any legal discovery team why IM and email are archived and you'll know what I mean.
Allen Varney
· 4 months ago
You're absolutely right that prediction markets can be uncannily accurate. In January 2007 I wrote about them (in the context of online games) in the article "Buzz Games" in the online gaming magazine The Escapist:
Thanks for the article. I certainly don't disagree that prediction markets are an important method of information gathering for any enterprise. However, I think we need more guidance, more best practices on how to use those results, when we should be casting for predictions, and just what sort of outcomes and circumstances we're encouraging predictions for. Prediction markets have an inherent limitation of being grounded in conventional thinking and of lowest common denominator thinking, even within an expertise community. We need to reconcile those limitations with its power to best identify when and how to use prediction markets. I would be the last person to deny the power of the effective use of prediction markets, but the issue I see is we seem to be implying, in evangelizing for prediction markets, that they are always accurate, when clearly they are not, or not as such, depending upon the point of time, the level of expertise of the crowd involved, and the visibility/circumstances of "today" which blind us to tomorrow.
The easiest example here of the limitation of predictive markets lies in the public space. Had we simply relied on predictive markets sometime around October or November 2008, those markets generally reduced McCain to no chance of getting the nomination as well as indicated Obama had little chance and by simplistic usage of predictive markets we'd say we would not see an Obama-McCain scenario. But any political analyst understood those results were rooted in the (mistaken and limited) conventional wisdom and general knowledge available of that moment and by larger and less sophisticated crowds in this knowledge space. And to be fair to the amateur analysts of that moment, even a crude predictive market among so-called pundits, I am pretty sure, would have shown similar results, even if some marked uptick for Obama and a marginal uptick (but still a call for a loss of the nomination) for McCain.
Now here also is where intelligent crowdsourcing can overcome/refine predictive market aggregation outcomes. Crowdsourcing mechanisms by which we identify experts with good track records and isolate the TREND among particular crowd elements could well inform us of particularities that would suggest the extreme flux of the moment and suggest the greater chances that McCain enjoyed all along along with the limited parameters but (at the time) remote-but-real probabilities/scenarios for an Obama breakthrough.
Similarly, in the economic prediction crowd, there was a growing trend before the economic crisis of people spotting a crisis coming, but it seems no prediction market came close to giving us an accurate picture that a meltdown was imminent. On the other hand, we did see a growing number of experts suggest exactly that in the 1-2 years preceding the meltdown. Again, a mixture of predictive markets but also careful trend monitoring of qualitative crowdsourcing feedback seems more useful here than purely predictive markets.
So we have to understand when, where, and how to use predictive markets and not turn those into tomorrow's surveys and polls, mechanisms which are too often abused and too often underexplained and become (mistakenly) too often dismissed. I suggest the greater frontier is that explanation rather than selling that they are useful.
bob roan
· 4 months ago
It sounds great, but I’m too confused to jump. Which way? What does a good application look like? Could a retailer use it to decide if a new product line would make sense? What size, in dollars, would be necessary? Or could an entrepreneur use it to check out ideas? Or is this just for big companies? And then, who do I ask to be part of the prediction markets? Employees, customers, the general public? The science is compelling, but how does a small business person figure out how to apply it?
glenn morgan
· 4 months ago
Enjoyed your post. 1) As a base contribution, “crowdcasting” provides additional data from which you may compare more traditional predictive models; if the two sets of predictions vary it is time to start digging into assumptions and weights associated with your traditional variables as well as the weight assigned to a costly error
2) Evaluations such as those provided by Crowdcast demonstrate the rational actor is joined in the marketplace by the rational crowd; assimilating data and optimizing likely outcomes via a much larger set of inputs and possible outcomes than those applied to an individual
3) Traditional predictions are often costly. Crowdcasting techniques are cost effective; at a minimum they provide a powerful directional starting point. Why would you not employ a cost effective technique to confirm your more traditional forecasts are marching in the correct direction?
Odzyskiwanie Danych
· 4 months ago
The crowd gives you statisticaly better chance of getting a right anwser. People who are wrong will simply be in minority. When asking "professionals" for opinions you're risking getting the wrong answer as "the only true one".
howtogrowtaller
· 4 months ago
that is very interesting, i actually learned something.
TimoteoManna
· 1 month ago
Go for a casual date. It is never cool to plan like you are about to make a marriage proposal when you are just blind date uncensored about to go to a blind date. A cozy café or park that allows lots of conversation and exchange of ideas will do the trick.
All Comments
The only reason I can see not to predict is when you're occupying a niche market and have your faithful customers.
In the Electronic Arts example, their front-line workers are often gamers (EA's target customer), so they serve as a good proxy. The often cited Best Buy "Blue Shirt Nation" example also holds to this. Best Buy employees are reasonable proxies for the electronics enthusiasts that shop at Best Buy stores.
Does this resonate with the data you're seeing? Do participants need to be proxies or "informed enough" to make valid decisions in these markets? Is the crowd more accurate even when making predictions outsides its collective expertise?
With government regulation looking like it'll only increase going forward (at least in the U.S.) it strikes me that corporations seeking to aggregate knowledge in the enterprise using PMs as a method will need to know "who knew what and when" more than ever before.
The Times piece and subsequent comments loiter over this point of anonymity as a primary asset and differentiator (which I acknowledge by the way), but the upside escapes me if it's impossible to answer the who-knew-when question.
Other than removing disincentives to participate where insider knowledge to game the market might prove compelling for the insider to remain anonymous, the reason regulators will seek to uncover them is to present a paper trail to call out these insiders in the event of impropriety. As any legal discovery team why IM and email are archived and you'll know what I mean.
http://www.escapistmagazine.com/articles/view/i...
The Oddhead blog by Dr. David Pennock, a computer scientist at Yahoo, frequently covers prediction markets:
http://blog.oddhead.com/
The easiest example here of the limitation of predictive markets lies in the public space. Had we simply relied on predictive markets sometime around October or November 2008, those markets generally reduced McCain to no chance of getting the nomination as well as indicated Obama had little chance and by simplistic usage of predictive markets we'd say we would not see an Obama-McCain scenario. But any political analyst understood those results were rooted in the (mistaken and limited) conventional wisdom and general knowledge available of that moment and by larger and less sophisticated crowds in this knowledge space. And to be fair to the amateur analysts of that moment, even a crude predictive market among so-called pundits, I am pretty sure, would have shown similar results, even if some marked uptick for Obama and a marginal uptick (but still a call for a loss of the nomination) for McCain.
Now here also is where intelligent crowdsourcing can overcome/refine predictive market aggregation outcomes. Crowdsourcing mechanisms by which we identify experts with good track records and isolate the TREND among particular crowd elements could well inform us of particularities that would suggest the extreme flux of the moment and suggest the greater chances that McCain enjoyed all along along with the limited parameters but (at the time) remote-but-real probabilities/scenarios for an Obama breakthrough.
Similarly, in the economic prediction crowd, there was a growing trend before the economic crisis of people spotting a crisis coming, but it seems no prediction market came close to giving us an accurate picture that a meltdown was imminent. On the other hand, we did see a growing number of experts suggest exactly that in the 1-2 years preceding the meltdown. Again, a mixture of predictive markets but also careful trend monitoring of qualitative crowdsourcing feedback seems more useful here than purely predictive markets.
So we have to understand when, where, and how to use predictive markets and not turn those into tomorrow's surveys and polls, mechanisms which are too often abused and too often underexplained and become (mistakenly) too often dismissed. I suggest the greater frontier is that explanation rather than selling that they are useful.
What does a good application look like? Could a retailer use it to decide if a new product line would make sense? What size, in dollars, would be necessary? Or could an entrepreneur use it to check out ideas? Or is this just for big companies? And then, who do I ask to be part of the prediction markets? Employees, customers, the general public?
The science is compelling, but how does a small business person figure out how to apply it?
1) As a base contribution, “crowdcasting” provides additional data from which you may compare more traditional predictive models; if the two sets of predictions vary it is time to start digging into assumptions and weights associated with your traditional variables as well as the weight assigned to a costly error
2) Evaluations such as those provided by Crowdcast demonstrate the rational actor is joined in the marketplace by the rational crowd; assimilating data and optimizing likely outcomes via a much larger set of inputs and possible outcomes than those applied to an individual
3) Traditional predictions are often costly. Crowdcasting techniques are cost effective; at a minimum they provide a powerful directional starting point. Why would you not employ a cost effective technique to confirm your more traditional forecasts are marching in the correct direction?