Stephen E. Arnold
Using Real Data to Mislead
Viewers of graphs, beware! Data visualization has been around for a very long time, but it has become ubiquitous since the onset of Big Data. Now, the Heap Data Blog warns us to pay closer attention in, “How to Lie with Data Visualization.” Illustrating his explanation with clear examples, writer Ravi Parikh outlines three common ways a graphic can be manipulated to present a picture that actually contradicts the data used to build it. The first is the truncated Y-axis. Parikh writes:
“One of the easiest ways to misrepresent your data is by messing with the y-axis of a bar graph, line graph, or scatter plot. In most cases, the y-axis ranges from 0 to a maximum value that encompasses the range of the data. However, sometimes we change the range to better highlight the differences. Taken to an extreme, this technique can make differences in data seem much larger than they are.”
The example here presents two charts on rising interest rates. On the first, the Y-axis ranges from 3.140% to 3.154% — a narrow range that makes the rise from 2008 to 2012 look quite dramatic. However, on the next chart the rise seems nigh non-existent; this one presents a more relevant span of 0.00% to 3.50% on the Y-axis.
Follow up on Robert Steele & Anonymous: Most Analysis Software Sucks — And Story of How Steele Correctly Called BSA Not Being Signed in Afghanistan
The usefulness of computer aids to intelligence analysis (“tools”) depends a good deal on what sort of ‘intelligence’ you are talking about. Intelligence is information that has been subjected to a process of research and analysis to determine its relative accuracy and relevance. When trying to determine if “analytic software” can help this process it is necessary to look at the kind of information that is being processed.
In the field of technical intelligence, i.e. SIGINT, there are a number of “tools” that are very useful. Most of these so-called tools are retrieval programs of various sorts that allow the analyst to manipulate the data in various useful ways and some of these capabilities go back over ten years ago (clustering and linking related bits if information and geographic displays using GIS). The most important unclassified technical advance impacting on analysis today is the availability of authentic data mining programs for the analyst. Data mining is NOT simple data retrieval, as many birdbrains claiming to speak for the IC appear to believe. Data mining proper uses a suite of sophisticated algorithms capably of detecting hidden patterns and trends, finding anomalies that may not be apparent, and even changing the original query structure to reflect retrieved information. Oracle has such a program based on the Oracle relational database that has been around in one form or another for at least 15 years. Data mining obviously would be effective against “big data.” The problem with all this is that these tools are designed to make research and analysis easier especially when dealing with large amounts of unevaluated information. As “anonymous” observed they cannot replace an engaged and target smart analyst.
Got Crowd? BE the Force!
I read your appraisal over several times. Essentially, in my opinion, your understanding of the problems continues to be on the mark and remarkably consistent over the last twenty or so years. Yet your work on both the process and products of intelligence is very high level and in this latest appraisal, as in your previous works, you leave it up to the imagination of the reader to figure out how to actually implement the ideas you so eloquently express. This I think is a mistake in that potential employers, impressed with you macro ideas, would be interested in how these ideas could be brought to the implementation stage. Attached is a supplement to your appraisal on collection and analysis.
In any event I hope that this finds you well and upbeat. You deserve a position that would reflect both your knowledge and your commitment to saving the IC from itself.
ROBERT STEELE: The time has indeed come to create an alternative to the existing system. I have started to work with a select group across the emergent M4IS2/OSE network, on a firm geospatial foundation. While many of my ideas have been mis-appropriated and corrupted over the past 20 years, no one has actually attempted to implement the coherent vision — sources, softwares, and services all in one, and this time around, all open source, all multi-everything. The PhD thesis, the School of Future-Oriented Hybrid Governance, and the World Brain Institute — and perhaps even the Open Source Agency as a non-US international body — are the beginning of my final twenty-year run. Intelligence with integrity. Something to contemplate.
Extention of Appraisal Details
Four More Short YouTubes Below the Fold