Data Mining, Kevin Bacon, and Able Danger

The National Journal's Shane Harris provides a remarkably complete account of ABLE DANGER and its predecessor programs. (As I would guess most readers of this blog already know, ABLE DANGER was an Army intelligence program, based around data-mining methods, that reportedly identified Mohammed Atta as early as winter 2000.)

As it happens, ABLE DANGER was not part of the "mainstream" IC, but was actually a military project run out of one of the Army's intelligence support organizations, the Information Dominance Center (itself a part of the Army's Intelligence and Security Command). Harris describes the IDC's data-mining program as something of a "maverick" project that managed to attract serious skeptics from traditional IC players at the FBI and CIA (and even the DoD's own DIA). Eventually, at the end of 1999, the IDC was tasked by USSOCOM to look at al Qaeda; this program was given the code name "ABLE DANGER." When one of the ABLE DANGER "searches" began turning up names of U.S. citizens (raising a whole host of serious legal issues, since the program remained a military intelligence initiative), the entire project was shut down and its 2.5-terabyte database deleted.

From a process perspective, ABLE DANGER was on the cutting edge of data-mining for the late 1990s:

The harvest "was a mile wide and an inch deep," Kleinsmith said. It included more than two terabytes of information, too vast an amount to provide specific targets. The IDC analysts could see the broad outlines of Al Qaeda, particularly its transformation from an idealistic movement into an operational network that could possibly inflict damage. Names, locations, and capabilities, and even the group's financial sources, were "coming together," Kleinsmith said. But the data set was still too big.

That didn't stop the analysts from trying to pare the information down. The former IDC employee said analysts played what they called "the Kevin Bacon game," referring to the popular notion that the prolific film actor can be linked to any other actor through no more than five people. (The game is based on the "six degrees of separation" theory that anyone on Earth can be linked to anyone else through five intermediaries.)

Skeptical insiders point out that vague "links" of this sort barely constitute indications of anything, and certainly don't amount to "warning." This sounds right; one should take any claims that ABLE DANGER could have prevented the 9/11 attacks with a large grain of salt. But methodologies for warning intelligence are by their nature controversial, and, quite frankly, have always divided the intelligence community. (Cynthia Grabo, in her illuminating text on strategic warning, describes in great detail the culture clash between traditionalist "order-of-battle" people and warning analysts throughout the Cold War.) This should not detract from the more important point: ABLE DANGER (and IDC projects like it) represented the creative application of new methods and technologies to a problem that had proven resistant to traditional approaches. Today, five years later, one hopes that the same sort of creative spirit is being cultivated within the intelligence community.

Sidenote: Recently I've been re-reading A Short Course in the Secret War by "Christopher Felix" (now known to be James McCargar). The tension between the traditional components of the IC and the military's various intelligence organizations seems to have a long history. (McCargar, of course, worked for one such organization ("The Pond") during the Hungarian operation.)

[Hat tip: The Strata-Sphere]