In reaction to a question from one of the members of the Foreknowledge Group (an eclectic closed group of people who attended 4Knowledge's courses) on what exactly the 4x4 and 5x5 information evaluation systems are the following:
The numbers refer to the different grading (1 to 4, 5 or 6) of a specific aspect of information and source evaluation. Usually we would talk about a 4x4 system, but it should actually be 2x4 - meaning we have 2 variables/dimensions that are graded by 4 weightings.
Depending on organisational protocols, you would usually grade the:
1) The source's Reliability: the index of the consistent quality of the source reporting the information: Determining factors will be credibility of source, access, etc. The x5 in this case is 1 or A) always reliable, 2 or B) mostly reliable, 3 or C)sometimes reliable 4 or D) Not reliable 5 or E) new source. If you use the x4 system, it will be 1 or A)Reliable (meaning mostly reliable - leaving out the always reliable) 2 or B)sometimes reliable 3 or C) not reliable and 4 or D)Reliability unknown.
2) The information's Validity: this weighs the accuracy or truth of the information supplied, no matter who supplied it. Issues determining it would be relevancy, timeousness, credibility, contextuality etc. The gradings will be 1 or A)Confirmed by other sources (Is this then automatically true?) 2 or B)Probably true 3 or C) possibly true 4 or D) Doubtful 5 or E)Improbable 6 or F) Cannot be judged. The x4 system here will be 1)Confirmed,2)Possibly True, 3)Untrue, to 4)Factuality Unknown
In addition, some intel organisations add their evaluation of the usability of the information i.t.o. the restriction and distribution of the information.
3) Restriction grading will be: restricted, unrestricted, confidential, secret, top secret
4) Distribution gradings will be: In organisation, outside organisation, specific internal clients, specific outside clients, no liaison with anyone.
The 5x5x5 system is used by the UK law enforcement in their National Intelligence Model which by law included the handling/distribution as the 3rd variable/dimension. There is also the 6x6 - The Admiralty or NATO system, mostly used in the military and national security intel. You would hear someone say "this is a B2" meaning the source is usually reliable and the information probably true:
RELIABILITY OF THE SOURCE
A - Completely reliable
B - Usually reliable
C - Fairly reliable
D - Not usually reliable
E - Unreliable
F - Reliability cannot be judged
CREDIBILITY OF INFORMATION
1 - Confirmed by other sources
2 - Probably true
3 - Possibly true
4 - Doubtful
5 - Improbable
6 - Truth cannot be judged
For most of us, the distinction between possible and probable is really only a few grades of truth, and everyone has a different weighting to those grades. I might say this is a C source, while the handler would feel it is B and my manager a D. In my own experience, the handler would mostly say it’s a B2 source, either to make himself and the source look better or just because he's too lazy to really interrogate new information his known source would give.
Source reliability opens a Pandora's Box of issues: subjectivity, deception, bias, judgment etc. This issue was at the heart of the US' WMD report that led to the Iraq invasion. Deception in intelligence analysis, like in all other social sciences, impact our understanding of the world. Sadly, scientific research on deception in intel has had little impact on the day-to-day activities of analysts. The US' ODNI's "What we mean when we say: an explanation of estimative language" is the ODNI's attempt to educate the intelligence consumer on the limitations of intelligence and is attached to each of their products.
This might be good example to follow for other intel organisations. Of course it will again be open for interpretation, but at least it is a starting point. Estimative language is a discipline on its own..
Good articles on source evaluation: http://www.virtualsalt.com/evalu8it.htm; http://www.professormcgill.com/blog/2008/09/20/source-analysis-and-the-twenty-five-questions-of-schum-and-morris/
http://www.scribd.com/doc/12837752/Evaluating-Intelligence-K-Wheaton
http://www.scribd.com/doc/2959133/VERBAL-PROBABILITY-EXPRESSIONS-IN-NATIONAL-INTELLIGENCE-ESTIMATES
http://www.afpa.org.au/get/402.pdf on source management
The numbers refer to the different grading (1 to 4, 5 or 6) of a specific aspect of information and source evaluation. Usually we would talk about a 4x4 system, but it should actually be 2x4 - meaning we have 2 variables/dimensions that are graded by 4 weightings.
Depending on organisational protocols, you would usually grade the:
1) The source's Reliability: the index of the consistent quality of the source reporting the information: Determining factors will be credibility of source, access, etc. The x5 in this case is 1 or A) always reliable, 2 or B) mostly reliable, 3 or C)sometimes reliable 4 or D) Not reliable 5 or E) new source. If you use the x4 system, it will be 1 or A)Reliable (meaning mostly reliable - leaving out the always reliable) 2 or B)sometimes reliable 3 or C) not reliable and 4 or D)Reliability unknown.
2) The information's Validity: this weighs the accuracy or truth of the information supplied, no matter who supplied it. Issues determining it would be relevancy, timeousness, credibility, contextuality etc. The gradings will be 1 or A)Confirmed by other sources (Is this then automatically true?) 2 or B)Probably true 3 or C) possibly true 4 or D) Doubtful 5 or E)Improbable 6 or F) Cannot be judged. The x4 system here will be 1)Confirmed,2)Possibly True, 3)Untrue, to 4)Factuality Unknown
In addition, some intel organisations add their evaluation of the usability of the information i.t.o. the restriction and distribution of the information.
3) Restriction grading will be: restricted, unrestricted, confidential, secret, top secret
4) Distribution gradings will be: In organisation, outside organisation, specific internal clients, specific outside clients, no liaison with anyone.
The 5x5x5 system is used by the UK law enforcement in their National Intelligence Model which by law included the handling/distribution as the 3rd variable/dimension. There is also the 6x6 - The Admiralty or NATO system, mostly used in the military and national security intel. You would hear someone say "this is a B2" meaning the source is usually reliable and the information probably true:
RELIABILITY OF THE SOURCE
A - Completely reliable
B - Usually reliable
C - Fairly reliable
D - Not usually reliable
E - Unreliable
F - Reliability cannot be judged
CREDIBILITY OF INFORMATION
1 - Confirmed by other sources
2 - Probably true
3 - Possibly true
4 - Doubtful
5 - Improbable
6 - Truth cannot be judged
For most of us, the distinction between possible and probable is really only a few grades of truth, and everyone has a different weighting to those grades. I might say this is a C source, while the handler would feel it is B and my manager a D. In my own experience, the handler would mostly say it’s a B2 source, either to make himself and the source look better or just because he's too lazy to really interrogate new information his known source would give.
Source reliability opens a Pandora's Box of issues: subjectivity, deception, bias, judgment etc. This issue was at the heart of the US' WMD report that led to the Iraq invasion. Deception in intelligence analysis, like in all other social sciences, impact our understanding of the world. Sadly, scientific research on deception in intel has had little impact on the day-to-day activities of analysts. The US' ODNI's "What we mean when we say: an explanation of estimative language" is the ODNI's attempt to educate the intelligence consumer on the limitations of intelligence and is attached to each of their products.
This might be good example to follow for other intel organisations. Of course it will again be open for interpretation, but at least it is a starting point. Estimative language is a discipline on its own..
Good articles on source evaluation: http://www.virtualsalt.com/evalu8it.htm; http://www.professormcgill.com/blog/2008/09/20/source-analysis-and-the-twenty-five-questions-of-schum-and-morris/
http://www.scribd.com/doc/12837752/Evaluating-Intelligence-K-Wheaton
http://www.scribd.com/doc/2959133/VERBAL-PROBABILITY-EXPRESSIONS-IN-NATIONAL-INTELLIGENCE-ESTIMATES
http://www.afpa.org.au/get/402.pdf on source management
Comments