Dr. Stephen Marrin
Stephen Marrin Post-revision draft18 July 2011. Original draft submitted to Intelligence and National Security on 4 February 2011. Accepted for publication on 24 May 2011 pending minor revision.
Evaluating the Quality of Intelligence Analysis: By What (Mis) Measure?<
Dr. Stephen Marrin is a Lecturer in the Centre for Intelligence and Security Studies at Brunel University in London. He previously served as an analyst with the Central Intelligence Agency and US Government Accountability Office. Dr. Marrin has written about many different aspects of intelligence analysis, including new analyst training at CIA‘s Sherman Kent School, the similarities and differences between intelligence analysis and medical diagnosis, and the professionalization of intelligence analysis. In 2004 the National Journal profiled him as one of the ten leading US experts on intelligence reform.
Abstract: Each of the criteria most frequently used to evaluate the quality of intelligence analysis has limitations and problems. When accuracy and surprise are employed as absolute standards, their use reflects unrealistic expectations of perfection and omniscience. Scholars have adjusted by exploring the use of a relative standard consisting of the ratio of success to failure, most frequently illustrated using the batting average analogy from baseball.Unfortunately even this relative standard is flawed in that there is no way to determine either what the batting average is or should be. Finally, a standard based on the decision makers’ perspective is sometimes used to evaluate the analytic product’s relevance and utility. But this metric, too, has significant limitations. In the end, there is no consensus as to which is the best criteria to use in evaluating analytic quality, reflecting the lack of consensus as to what the actual purpose of intelligence analysis is or should be.
Evaluating the Quality of Intelligence Analysis: By What (Mis) Measure?
Evaluating the quality of intelligence analysis is not a simple matter. Frequently quality is defined not by its presence but rather by its absence. When what are popularly known as intelligence failures occur, sometimes attention focuses on flaws in intelligence analysis as a contributing factor to that failure.
Read full paper online.
Read the rest of this entry »
The declassified version of my 1968 Studies in Intelligence article can be found in a couple of different formats under “Jack Davis, Bogotazo.” Please post it. I think this is my first published work.
Take a look at the conclusions of an article I wrote 43 years ago, and substitute “North Africa” for “Latin America” and “al Qa’ida” for “Communists.”
APPROVED FOR RELEASE 1994 CIA HISTORICAL REVIEW PROGRAM 2 JULY 96
Distant events shape the craft of intelligence.
On the afternoon of 9 April 1948, angry mobs suddenly and swiftly reduced the main streets of Bogotá to a smoking ruin. Radio broadcasts, at times with unmistakable Communist content, called for the overthrow of the Colombian government and of “Yankee Imperialism.” Many rioters wore red arm bands; some waved banners emblazoned with the hammer-and-sickle. A mob gutted the main floor of the Capitola Nacional, disrupting the deliberations of the Ninth International Conference of American States and forcing Secretary of State Marshall and the other delegates to take cover. The army regained control of the city over the next day or two. But not before several thousand Colombians had been killed. It was the bogotazo.
Read entire analytic piece now declassified.
Others, notably Dr. Loch Johnson, de facto dean of the intelligence scholars in the English language, have explored both definitions and concepts for a theory of intelligence. Others, such as Jack Davis, have done much in the area of analytic tradecraft or the “art” of intelligence analysis (to match the “art” in clandestine operations and covert action).
Now the time has come to develop a science of intelligence. The first casualty must of necessity be the obsession with secret sources and methods, secret agencies, and secret clients. Intelligence is about decision-support, plain and simple, and the new science of intelligence will be developed along the lines of the services science developed by Dr. Jim Spohrer of IBM, and others. Dr. Spohrer provided the below in an email exchange today:
(1) “you can have a science of anything, if a community agrees it is important”
(2) “innovations that are based on sciences, not just management and engineering practice, can be advanced more systematically”
(3) “industry cares about innovation acceleration, can academia deliver a science? does the engineering and management exist in practice”
(4) “academia said we can establish a research area to help build the science under the engineering and management practice.”
A round-table is being formed and a new article will result.
Misappropriated without Attrribution–Free Online
, March 4, 2010
One Star for lack of ethics on the part of the publisher. Beyond five stars for content, free online as with all of Jack Davis’s stuff. Upgraded to 3 stars for proper pricing (after Amazon’s cut, publisher only makes roughly 3 dollars per book, which is totally fair).
This product was misappropriated from Jack Davis, dean of the intelligence analysis scholar-practitioners. While materials created within the US Government by US Government employees are generally not copyrighted because the taxpayer funded their creation, they are a) available free online; and b) generally considered off-limits to sleaze-bag publishers that troll for stuff (this happens to all of us, in my case with my monographs for the Strategic Studies Institute (SSI), all free online).
It’s nice that Jack’s work is respected and made available on Amazon, a truly global service.
It is very troubling that Jack Davis, who just asked me to find out who did this, has not been contacted by the publisher and offered both courtesy copies of his own work, and some modest recognition.
Read the rest of this entry »
Although this search brings up two relevant hits, Jack Davis, de facto dean of intelligence analytic tradecraft, is one of perhaps ten people at CIA that we absolutely hold in the highest regard. Here are all of the links, the first one being the specific report you are looking for. Searching on the web for “analytic tradecraft” is also interesting. Here it is better to search for <jack davis>.
1997 Davis A Compendium of Analytic Tradecraft Notes
Who’s Who in Public Intelligence: Jack Davis
2003 Davis (US) Analytic Paradoxes: Can Open Source Intelligence Help?
2000 PRIMER on Open Sources & Methods
Graphic: The UN and the Eight Tribes of Intelligence
Search: The Future of OSINT [is M4IS2-Multinational]
Review: Assessing the Tradecraft of Intelligence Analysis
Worth a Look: Book Reviews on Analysis
The UN and the Eight Tribes of Intelligence
This is Figure 2 in Chapter 1 of INTELLIGENCE for EARTH: Clarity, Diversity, Integrity, & Sustainability. The 20% that the UN receives today is based on UN input associated with our creation of the Class Before One Briefing, the UN slide as provided to Robert Steele can be found therein. For other UN intelligence training materials, see UN Intelligence Training.
The original conceptual depictions of “competing influences” on individual decision-makers were first developed by Dr. Greg Treverton teaching the Intelligence Policy Seminar at JIF School of Government, and Jack Davis, dean of the U.S. Intelligence Community scholar-analysts. The “eight tribes” (previously seven) are original to Robert Steele. Steele’s adaptation of Davis-Treverton first appeared as Figure 17 on page 53 of ON INTELLIGENCE: Spies and Secrecy in an Open World (AFCEA, 2000).