The Fortune-Teller Dilemma: Mastering CI/MI in the Age of AI
Every experienced Competitive and Market Intelligence (CI/MI) practitioner has seen it: by Thorsten Bill
You ask a frontier AI model to analyze a market threat, and it hands you a brilliant, 30-page deliverable. The prose is fluent, the structure is clean, and the analytical tables look ready for the board.
Then you audit Footnote 14 and realize the cited market study or primary APA reference simply does not exist.
For senior analysts, that single hallucinated reference kills the report on the spot. It is why many experienced intelligence professionals remain deeply skeptical of AI. One fake citation in front of the board destroys your professional credibility.
But a fake citation is actually the easy failure mode. It gets caught because it is obvious. The true threat isn't the hallucinated reference you do catch; it is the methodologically fluent fallacy on page 4 that you don't catch because the rest of the text reads like expert prose.
When an output looks structurally complete, human reviewers naturally lower their critical guard. A minor, unexamined assumption on page 4 silently propagates into a multi-million-dollar strategic bet on page 30.
To avoid operating a high-tech fortune-telling machine, intelligence practitioners must master a fundamental distinction: procedural rigor is not epistemic rigor.
The Fundamental Structural Contradiction in LLMs
Dismissing AI errors as the simple work of a "stochastic parrot" misses the real operational hazard in strategic and especially high-stakes—intelligence.
Large language models do not fail because they are "dumb"; they fail because of a core architecture conflict when applied to open-world systems:
- The Engine is Probabilistic: Under the hood, an LLM predicts the next most plausible token based on statistical weights.
- The Output is Framed as Deterministic: Instead of delivering explicit confidence intervals or dynamic scenario ranges, the model outputs a smooth, single narrative formatted as absolute fact.
- The Environment is Non-Deterministic: Real market dynamics are volatile, adversarial, and non-deterministic.
In engineering, you can simply use a test bench to catch bugs or flaws before a product launch. In business strategy, there is no pre-execution test bench. The market is the test bench, and you only get to run the test once after committing real capital.
When a human analyst cuts a corner under time pressure, they retain a metacognitive "gut feeling" of discomfort. A mental warning light that says, "I am leaning on a thin assumption here; we need to monitor this." An LLM possesses no metacognition.
An LLM cannot bear to have "no answer." Bound by its objective function, it takes silent cognitive shortcuts, covers them with flawless prose, and presents an unverified guess with the exact same polished authority as a verified metric.
Two Best Practices Drawn from Our Own Discipline
AI researchers list hundreds of fundamental AI fallacies, bias issues, algorithmic flaws, and structural problems in LLMs, continually trying to patch them. However, as CI/MI practitioners, we do not need to wait for computer science to solve this. We can apply two fundamental best practices drawn directly from our own discipline:
- Stick to established Structured Analytic Techniques (SATs): Decompose the technique into steps, explicitly identifying which parts AI can automate and which parts it cannot.
- Learn to identify AI fallacies using existing CI/MI skills: You don't need a deep degree in LLM architecture. Most AI fallacies are identical to traditional analytical fallacies, cognitive biases, blind spots, missing verification steps, or blunt shortcuts.
You will find these exact fallacies in AI reports just as you would in a flawed human analysis. Simply speaking, it is a matter of telling apart a presumptuous draft from a solid analysis.
How to Mold Your AI Analysis into a Fixed Form
To achieve consistent, repeatable, and verifiable analysis quality, you must first define exactly how the analysis is performed. Simply put: you must enforce the Structured Analytic Technique.
If you leave methodology to AI discretion and merely state a question without defining how to build the answer, the AI will default to a generic answer and write a 30-page report around it without hesitation.
Human-in-the-Loop Decomposition: The Mechanical vs. Judgment Boundary
Once you identify the appropriate analytic technique, decompose it into individual steps and split them into mechanical steps (which AI can automate) and judgment-laden steps.
Be aware that automated steps always require an audit trail to verify correctness. Judgment-laden steps include verification checkpoints, but they also encompass any step requiring an executive decision such as selecting the analysis method in the first place or assessing organizational risk.
In AI research, this principle is called Human-in-the-Loop (HITL). Consider this decomposition as a meta-framework to embed HITL checkmarks into your AI-supported workflow before you execute it.
| Dimension | Mechanical Steps (Delegate & Verify) | Judgment-Laden Steps (Human-Owned) |
|---|---|---|
| Definition | Steps whose correctness can be verified against a rule, computation, or external data source without situated business context. | Steps that depend on domain expertise, organizational context, and risk tolerance. |
| AI’s Role | Brainstorming candidate lists, cross-referencing datasets, computing consistency matrices, and drafting initial text fast. | Defining system boundaries, weighting factor plausibility, and evaluating source credibility or deceptive intent. |
| The Rule | Safe to automate if it produces an inspectable, checkable artifact (e.g., a source list or raw matrix). | Never automate. The analyst must explicitly make, justify, and own these calls. |
Rule of thumb: If you are ever unsure whether a step is mechanical or judgment-laden, treat it as judgment-laden. Overclaiming a judgment call as mechanical is how an unexamined guess ends up hidden inside a checked-looking table.
Uncovering AI Fallacies: Beyond Simple Prompts
Mastering AI in intelligence work is not just about finding a "magic prompt recipe for your Structured Analytic Technique." It requires applying classical intelligence discipline to human-AI dialogue.
Take Jan Herring’s Key Intelligence Questions (KIQ) framework as an example: decision-makers rarely state their true strategic need in their initial request. Similarly, users naturally submit brief, under-contextualized prompts. Because AI is engineered to be helpful, it defaults to sycophancy, generating a fluent answer to the literal question while missing the unstated strategic agenda.
Conversational AI Literacy in Practice
Human-in-the-Loop execution is, to a large extent, the art of finding fallacies in an AI report or conversation. Experienced CI/MI practitioners do not need to learn this skill from scratch, it mirrors HUMINT elicitation and source evaluation.
Here are four practical techniques to uncover AI fallacies using conversational literacy:
- Pre-Execution Interviewing: Run a multi-turn elicitation phase to define the analysis boundaries and clarify methodology before asking the model for a final draft.
- Re-Anchoring Context: Periodically repeat established ground rules and analysis steps, summarizing intermediate results to prevent conversational drift across long contexts.
- Opening Meta-Layers: Pause execution to explicitly evaluate a specific analytical step. For example:
- "List the contextual assumptions and hypotheses you are making about our market."
- "Perform a Jekyll & Hyde / Red Team analysis to challenge this assumption."
- "Adopt the persona of the customer: What are their true unstated needs and pain points?"
- Recognizing a Broken Conversation: AI conversations can irreversibly collapse due to context limits, attention degradation, or creative loop traps. It is like working with a colleague who loses productivity from cognitive overload. When this happens, simply save your verified intermediate artifacts and start a fresh session.
Summary
In high-stakes strategy, accepting a probabilistic report as deterministic fact isn't intelligence work. It is gambling.
The true value of the intelligence practitioner is not generating reports faster, but enforcing the Human-in-the-Loop guardrails that keep analysis grounded in real-world truth. Verifying veracity, extracting ground truth from AI narratives, and preventing fast-path shortcuts is an elicitation discipline. It requires persistent skepticism, cross-checking sources, and rigorously separating ground truth from unexamined assumptions.
Next Steps: Moving from Theory to Execution
To help practitioners operationalize these principles, we have formalized the HITL Decomposer Framework into a practical, step-by-step implementation guide and white paper. It includes a 4-point enterprise prerequisite checklist, a 5-column classification system, and an executable system prompt that automatically splits any Structured Analytic Technique into verifiable mechanical steps and human-owned judgment calls.
👉 Read the full white paper: Human-in-the-Loop Decomposer: An Operational Design Pattern for AI-Assisted Strategic Intelligence