On the night before the 1986 Challenger launch, engineers at Morton Thiokol warned that the shuttle’s O-rings would perform worse in the cold. The forecast temperature was 26 to 29 degrees Fahrenheit, far below anything the shuttle had flown in before. The warning reached the decision makers as slides, and the slides flattened what the engineers knew.
Deniz Tunçalp’s new paper in the Journal of Management History argues that organizations are repeating this pattern with generative AI. The technology is new and the pressure behind it is old. Managers want briefings that are short and ready to act on, and any tool that delivers them quietly decides what the briefing leaves out.
What went wrong with the slides
The paper draws on the Rogers Commission report, Diane Vaughan’s study of NASA and Edward Tufte’s analysis of the slides themselves. A bullet-point slide could not carry probabilistic, non-linear reasoning, so the engineers appeared to lack definitive evidence. The deck of 13 charts also left out temperature data for flights with no O-ring damage. That removed the comparison that would have shown the link between cold and failure. The burden of proof moved too. Managers no longer had to show that the launch was safe, and engineers had to prove it was dangerous.
The same pattern with AI
Start with how much a model can read at once. When a document is longer than the model’s context window, many systems read the first part and still write a confident summary, with no note about what they skipped. Even inside the limit, research finds that models handle information in the middle of long inputs less well. The paper calls this token-based epistemic filtering.
Then there is the expert’s place in the process. In Mata v. Avianca, a lawyer used ChatGPT to research case law and filed a brief with six fictitious citations. The paper calls this algorithmic epistemic substitution, which happens when the output is treated as the final answer and no expert checks it.
Finally, AI tools sit inside processes where errors travel fast. Following Charles Perrow’s normal accident theory, the paper argues that complex, tightly coupled systems will produce some failures whatever management intends.
How the cycle builds
The paper’s model is a loop. Pressure for simplicity leads an organization to adopt a tool that simplifies. The tool filters, and its output looks complete and confident. Decisions based on it work often enough that people stop checking, which Vaughan called the normalization of deviance at NASA. Reliance grows. When a failure finally comes, it looks unprecedented, even though its cause was built into the setup. Benefits show up at once while risks stay hidden, so organizations keep adopting even when they know the history.
Where the argument applies
The argument is conditional. It applies most strongly to decisions that carry high stakes, depend on expertise and come under time pressure, such as healthcare, law, finance and safety-critical engineering. For routine tasks, brainstorming and easy-to-check questions, the paper says the benefits may outweigh the risks. It is also a conceptual study, so the model still needs empirical testing.
What to do about it
The paper proposes what it calls epistemic resilience practices. Scale the checking to the stakes, to how hard the decision is to undo, to how much it depends on specialist knowledge and to how opaque the system’s coverage is. When a tool cannot confirm that it covered the whole input, treat its output as a partial sample and have an expert review it. Expect systems to disclose which parts of the input they processed. Set up red teams of independent domain experts. Design workflows where AI drafts, experts review and annotate, the team discusses what the AI may have missed, and the final decision combines both. Keep audit trails of how often summaries were used and whether anyone verified them.
The paper does not call for avoiding generative AI. Its goal is to make the structural dynamics visible so organizations can act on them. Its closing line puts the stakes briefly. When lives, justice and organizational viability depend on decisions, “there are no shortcuts to knowing well.”
Read the full paper, Historical failures and epistemic accidents, in the Journal of Management History. If your team is adopting AI tools and wants to decide where verification belongs, get in touch.