The U.S. Intelligence Community is considering using crowdsourcing as a means of predicting at least some global occurrences including geopolitical, security, and economic events.
The initiative, called the Geopolitical Forecasting Challenge, is part of a new contest of sorts sponsored by the Intelligence Advanced Research Projects Activity, an agency under the Office of the Director of National Intelligence that is charged with leading research to overcome difficult challenges relevant to the overall intelligence community.
“What we’re going to do here is allow anyone who’s interested – citizen scientists, hobbyists – to develop forecasting methods to attempt to anticipate what events are going to occur sort of in real time,” said Seth Goldstein, a program manager with IARPA, in an interview with an American media outlet.
Goldstein explained how the initiative might work.
“For instance, will the United Nations vote to impose new sanctions on some particular country, say, by the end of the year? … We might ask that question about multiple countries. Or will any additional EU country vote to depart the EU by the end of the year?” he said, adding “We might ask that question about multiple countries. Or will any additional EU country vote to depart the EU by the end of the year?”
“There’s $200,000 in prize money at stake,” Goldstein said. “We’re going to be providing them with state-of-the-art data on crowdsourced human judgments. And we know it’s the state of the art because we’ve validated this state of the art in a previous IARPA research program called ACE (Aggregative Contingent Estimation).”
“So this program was about improving on what was then a very limited state of the art in crowdsourced forecasting. We achieved that in that program. We beat the existing state of the art by greater than 50 percent,” he added.
When asked why he thought that a crowd of individuals would be a more accurate forecaster of coming events than a dedicated expert with a classified security clearance and all the data of the U.S. intelligence community and his or her disposal, Goldstein said the idea is not to completely replace subject matter experts (SMEs) and analysts.
“The problem with expert judgment generally is that it’s difficult to know in advance which expert is going to make a correct forecast on any particular event,” he said. “Somebody might get it one time and they might get it wrong another time. The idea behind crowdsourcing is that if you assemble a reasonably sized crowd, a large crowd of hundreds or even thousands of people making judgments, the idea of there being any particular directional bias in some aggregate of that judgment is reduced. It’s not to say experts can’t make accurate forecasts. It’s to say if you had to choose a method, this might serve you better.”
But, he added, “expert judgment is always going to be innately valuable. Experts provide critical context about causal understandings of current phenomena, past phenomena and so forth. So, no, I don’t think that we’re going to put sort of the intelligence analysts out of business. [source]
Analysis: This agency is tasked with ‘outside the box’ thinking; it will be interesting to see how this project works in the coming months and years. That said, it’s hard to imagine an intel community without dedicated SMEs who possess full knowledge of and historical data on, say, North Korea or China.