Hypothesis Testing

The human mind works with a confirmation bias: the propensity to seek out evidence that supports our initial intuitive solution to a problem. A technique developed at the CIA to help analysts rigorously challenge evidence and review it in light of multiple explanations or hypotheses is popular with analysts working in law enforcement, the military, and the intelligence community.

Now anyone investigating cause and effect relationships, exploring potential answers to problems, or making a decision can benefit from an extremely simple-to-use structuring tool called Hypothesis Testing.

If someone has evidence and a hypothesis, they can use this app to structure there thinking and help explain it to others. For example, if you are considering your investment portfolio and you’re initial hypothesis is that the market is about to take off, you will no doubt become biased to look for evidence in the news that things are beginning to look up: earnings reports may be up in a particular sector, peace may be near in the Middle East, etc. – and it will become easy for you to filter that evidence.

What this app does is allows you to enter your hypothesis (like “bull market”) but also to enter alternative hypotheses (like “bear market” and “static market”). Then you can enter all the evidence as you collect it: earnings reports; peace; energy reserves; weather, and so on. To spur your thinking to consider alternative hypotheses, the app even challenges you by asking Alexander’s Question: What new information would be sufficient to change your decision? Simply asking this question often provokes investigators to seek out new evidence to “disprove” a favorite hypothesis, which is the hallmark of a critical thinker.

After the evidence is collected and entered into the app, the app then presents one piece of evidence at a time in a matrix with each of the competing hypotheses. “Earnings reports up in the tech sector” may be shown along with “bear, bull, and static market” hypotheses. The app then requires that you use a C, I, or ? to related the evidence to each hypothesis: the C means that the evidence is consistent with the hypothesis. In our example, earnings reports up in the tech sector is consistent with a bull market, so we’d use a C.

The same piece of evidence is not inconsistent with a bear or static market, however – and here’s the critical thinking required – so we’d need to place a C with the other hypotheses as well, indicating that this one piece of data is consistent with a number of hypotheses. The evidence of earning being up in one sector is simply not specific enough to indicate where the entire market is heading.

While a lot of this type of critical thinking takes place in our heads, the app keeps track of it all. When all of the evidence is in, the app scores the hypotheses, and lists the hypothesis with the least amount of inconsistent evidence as being the most likely. In a sense, we can never “prove” a hypothesis to be true, but instead we “disprove” all of our hypotheses except one, and that one is the most likely. As Sherlock Holmes said, “"How often have I said to you that when you have eliminated the impossible, whatever remains, however improbable, must be the truth?”