The Human Authority Problem

AI can accelerate what institutions know. It must not accelerate what they are willing to act upon.

On September 18, 2026, CNN reported an extraordinary close call inside the United States military. Earlier in the spring, while the war with Iran was underway, an intelligence report circulated through the military concluding that a Chinese vessel transiting the Middle East was carrying components associated with a nuclear weapons program. The conclusion was serious enough to trigger preparations for an interception. Military aircraft were already in the air, and armed personnel were preparing for the possibility of boarding the vessel.¹

The intelligence was wrong.

A Special Operations Command analyst had supplied an artificial intelligence chatbot with a combination of classified signals intelligence and open-source information, including information about the vessel’s manifest. The system incorrectly identified its cargo as connected to a nuclear weapons program. AI was subsequently used in producing the intelligence report that entered military distribution. Only shortly before the planned operation did experienced subject-matter analysts examine the underlying information closely enough to identify the error. The operation was called off.²

CNN was unable to determine whether the chatbot involved was a commercial product or an internal government AI system. Nor does the available reporting establish what the vessel was actually carrying. What it does establish, based on multiple sources familiar with the incident, is that an AI-assisted analytical error traveled far enough through an institutional intelligence process to contribute to preparations for military action involving a vessel belonging to another major power.³

The episode is easy to characterize as another example of artificial intelligence hallucinating.

That interpretation misses the more consequential failure.

AI systems produce erroneous information. Humans do too. Neither fact is particularly revelatory. What makes this episode significant is that an unverified machine-derived conclusion apparently traveled through a trusted institutional system far enough to influence preparations for action.

The problem was not simply that AI was wrong.

The problem was that being wrong did not stop the information from acquiring authority.

The Value of Speed

There are compelling reasons for the military to use artificial intelligence.

Modern intelligence operations confront quantities of information that can exceed the practical analytical capacity of human teams. AI systems can rapidly correlate disparate information, detect patterns, identify anomalies, generate hypotheses, summarize large collections of material, and help analysts interrogate complex environments. Military scholarship has identified faster analysis, improved situational awareness, threat identification, planning, and decision support among the potential benefits of incorporating AI into military operations.⁴

That distinction matters because military decisions sometimes have to be made extraordinarily quickly. Speed can save lives. Faster identification of a threat can provide commanders with additional options rather than fewer. Artificial intelligence can therefore extend human analytical capability in ways that would be irresponsible to dismiss simply because the technology remains imperfect.

Military planning also takes place in an environment characterized by incomplete or inaccurate information, uncertainty, constrained resources, and limited time. AI can help evaluate courses of action, anticipate possible adversary behavior, assess risk, estimate resource requirements, and synchronize operations.⁵

The attraction is therefore not mysterious.

But speed creates its own governance problem.

The faster an institution becomes capable of producing analysis, the greater the temptation—and eventually the organizational expectation—to produce decisions at the same speed.

Those are not the same thing.

AI may accelerate analysis. It must not accelerate authority.

From Machine Output to Institutional Knowledge

The distinction becomes especially important when AI-generated information leaves the interface in which it was created.

A chatbot response looks like a chatbot response. Its origin is obvious. A responsible user encountering an unexpected conclusion can inspect the underlying sources, challenge the reasoning, request alternative interpretations, or independently reproduce the analysis.

But information changes when it enters an institution.

In the Chinese vessel incident, AI was not confined to an isolated analytical exchange. The erroneous conclusion became part of an intelligence product that was disseminated through the military. More experienced analysts examined the underlying information only later, when preparations for the operation were already well underway.⁶

That transformation deserves attention.

The recipient is no longer looking at a chatbot response.

The recipient is looking at an intelligence report.

Formatting, institutional vocabulary, classification markings, established distribution channels, and organizational context can confer credibility independently of the quality of the reasoning underneath them. Uncertainty generated by a machine can therefore emerge from an institutional process wearing the appearance of established knowledge.

Call this institutional camouflage.

It does not require deception. Nobody has to deliberately disguise machine-generated information. The institution itself can perform the transformation simply by processing uncertain information through mechanisms historically associated with authority.

This creates a new requirement for AI governance:

Provenance must survive institutionalization.

The Provenance Problem

The Department of Defense already recognizes traceability as a foundational principle of responsible artificial intelligence. Its Responsible Artificial Intelligence Strategy and Implementation Pathway identifies five ethical principles—Responsible, Equitable, Traceable, Reliable, and Governable. Traceability requires transparent and auditable methodologies, data sources, design procedures, and documentation. Responsibility requires personnel to exercise appropriate judgment and care while remaining responsible for the development, deployment, and use of AI capabilities.⁷

The Department describes the desired outcome not as unquestioning confidence in AI, but as “justified confidence.” Its implementation framework calls for testing, evaluation, verification, validation, monitoring, and an understanding of system limitations and failure modes.⁸

More recent military-AI scholarship reaches a similar conclusion. Governance frameworks for military AI emphasize data provenance, audit trails, validation procedures, escalation controls, and retained human decision authority in high-risk applications. They also identify a deeper institutional effect of AI adoption: it can redistribute authority between humans and machines.⁹

But provenance becomes substantially more complicated when AI contributes not merely to a document, but to individual assertions within it.

Imagine that an AI-generated claim enters an intelligence report without its evidentiary lineage remaining visible. That report becomes part of the institutional record. A subsequent analyst encounters it and reasonably treats the earlier report as intelligence. A later AI system retrieves the same material while performing another analysis. Eventually, the original machine-generated assertion may appear to have multiple sources supporting it when those sources ultimately descend from the same erroneous inference.

The pathway can become:

AI-generated error → accepted intelligence → institutional record → subsequent analysis → apparent corroboration

The original uncertainty has been laundered through the institution.

This might be called recursive error laundering: an error gradually acquiring credibility as its machine origin and evidentiary lineage become increasingly distant from the people or systems encountering it.

Document-level disclosure that “AI was used” may therefore be insufficient in high-consequence environments. What matters is the provenance of consequential assertions: where a claim originated, what evidence supports it, what portion represents observation rather than inference, whether AI contributed to that inference, and whether the conclusion has been independently verified.

Without that lineage, institutions risk creating synthetic corroboration from their own previous mistakes.

A Human in Which Loop?

The conventional answer to consequential AI decision-making has often been reassuringly simple:

Keep a human in the loop.

But the phrase obscures an important question.

What is the human actually doing there?

A human who clicks approve is in the loop.

A human who reads an AI-generated summary without inspecting its sources is in the loop.

A supervisor who assumes an analyst verified the machine’s reasoning upstream is in the loop.

None necessarily constitutes meaningful human agency.

Research on human-AI collaboration provides reason to take that distinction seriously. Automation bias occurs when teams accept AI recommendations uncritically even when contextual information suggests the system may be wrong. The opposite failure, algorithm aversion, occurs when potentially useful AI output is rejected too readily. Effective human-AI collaboration therefore depends not on maximizing trust, but on calibrating it—maintaining enough confidence to exploit AI capability while preserving enough skepticism to invoke independent human judgment when circumstances warrant it.¹⁰

That balance is not automatically produced by putting humans and machines together. A synthesis drawing on 106 human-AI experiments found that hybrid teams often fail to outperform the better of humans or AI operating independently. A significant problem is coordination: teams frequently lack explicit rules governing when machine recommendations should be accepted, verified, challenged, or overridden.¹¹

The implications become particularly important as consequences increase. One proposed trust-calibration framework varies the appropriate human-AI relationship according to the cost of error and the kind of knowledge required. For low-cost, explicit tasks, automation may reasonably take the lead. For explicit decisions carrying high costs, the relationship becomes AI proposes, human verifies. For high-stakes decisions requiring contextual or tacit knowledge, it becomes human first, AI assists.¹²

That is considerably more meaningful than requiring a human somewhere in the workflow.

Human presence is not necessarily human agency.

Meaningful agency requires the ability—and, in consequential situations, the obligation—to interrogate machine output. What are the primary sources? What does each source actually establish? Which claims are observations and which are inferences? What assumptions were introduced during analysis? What contradictory evidence exists? Can the conclusion be independently reproduced? Where did each consequential assertion originate?

And who is accountable for deciding that the evidence is sufficient to act?

These are not arguments against AI.

They are mechanisms for using it responsibly.

Trust Is Not Verification

There is another difficulty.

Humans do not encounter AI output neutrally.

Repeated exposure to AI advisory systems can produce overreliance, while poorly defined responsibility for reviewing machine contributions can allow accountability to diffuse across a team. Explicit role definition—what AI proposes, what humans must verify, and who ultimately authorizes a decision—is therefore itself a governance mechanism.¹³

That distinction becomes even more important inside trusted institutions.

An analyst may appropriately distrust an unsupported chatbot assertion while trusting an intelligence report containing essentially the same assertion. The information has not necessarily become more reliable.

Its container has become more authoritative.

Institutional processes must therefore be designed not merely to establish trust, but to preserve the reasons for that trust.

This distinction is consistent with the Department of Defense’s own Responsible AI framework. Traceability, reliability, governability, and human responsibility are not merely abstract ethical aspirations. They are mechanisms through which confidence can remain connected to evidence. The Department’s current AI-readiness guidance makes the accountability issue explicit, asking organizations whether a clear line of accountability exists for AI-generated outputs and decisions.¹⁴

That question should become more important—not less—as AI systems improve.

The Verification Burden

Not every AI output warrants the same scrutiny.

Requiring independent expert verification of every machine-generated scheduling suggestion, budget summary, or routine administrative classification would eliminate much of the efficiency AI is supposed to create.

Consequences matter.

A useful governance principle is therefore proportional:

The greater the consequence of error, the greater the verification burden before AI-derived information acquires institutional authority.

Research on military AI identifies precisely this tension. Faster decision support can improve operational effectiveness while simultaneously creating risks from false positives, opacity, overreliance, and decisions made without sufficient understanding of how a recommendation was produced. High-risk uses therefore require greater human intervention, explainability, and accountability than routine applications.¹⁵

The same principle appears in contemporary human-AI research. Safety-critical environments warrant conservative reliance and redundant checks, while lower-stakes, high-frequency tasks can tolerate greater automation.¹⁶

The Department of Defense itself adopts a risk-sensitive approach. Its implementation pathway explicitly rejects rigid, one-size-fits-all governance and distinguishes basic AI research from capabilities approaching operational deployment. An AI capability ready for use in an operational system must demonstrate compliance with applicable safety and security standards, while the Department seeks to balance responsibility with speed and ease of implementation.¹⁷

This suggests that the critical governance boundary is not necessarily where AI enters a process.

It is where its output becomes actionable.

Before consequential machine-derived information crosses that boundary, the institution should be able to establish provenance, characterize uncertainty, distinguish observation from inference, independently verify critical claims, and identify an accountable human decision-maker.

The Chinese vessel incident illustrates why that boundary matters. Military aircraft were already airborne and armed personnel were preparing for the possibility of boarding the vessel when experienced subject-matter analysts examined the underlying information and discovered that the AI-assisted conclusion was wrong.¹⁸

Verification occurred.

It simply occurred dangerously late.

Faster Than Verification

Perhaps the most consequential characteristic of artificial intelligence is not intelligence at all.

It is acceleration.

AI can compress the interval between information gathering, interpretation, analysis, production, and decision-making. That compression can create enormous strategic advantages. But an institution’s verification mechanisms do not automatically accelerate at the same rate.

That creates an asymmetry.

A system capable of producing ten times as much analysis does not automatically create ten times as many experienced analysts capable of validating it. Faster intelligence production can therefore create pressure on the very humans intended to govern it.

The reported Chinese vessel incident may not have been an isolated manifestation of that problem. Sources familiar with military AI use told CNN that some AI-assisted intelligence products have reached operational use without being fully vetted, while AI use in targeting operations has been increasing. The reporting also described inconsistent practices across military organizations and concerns about the absence of sufficiently consistent guidance governing how human involvement should operate in consequential applications.¹⁹

One source captured the acceleration problem particularly well:

“AI allows you to get to a bad idea faster.”²⁰

That is the paradox.

The strategic attraction of AI is speed and scale.

Its failure modes inherit those same properties.

Epistemic Due Process

The challenge, then, is not merely keeping humans involved.

It is preserving the institutional processes through which knowledge earns the authority to produce action.

AI compresses analytical time. Institutions must resist allowing it to compress epistemic due process by the same amount.

Uncertainty should remain uncertainty as information moves through a system. An inference should remain identifiable as an inference. Low-confidence information should not acquire high-confidence presentation merely because it has passed through an authoritative document template. Machine-generated conclusions should retain evidentiary lineage even after they become part of an institutional record.

And consequential assertions should not become more certain simply because they have traveled farther.

This is where agility and discipline must coexist. Institutions that fail to exploit AI may eventually become too slow for the environments in which they operate. But agility without epistemic discipline can become dangerous acceleration.

The objective is not to make AI slower.

It is to ensure that verification remains capable of catching up.

Human Authority

Humans produce bad intelligence. They misunderstand evidence, succumb to cognitive biases, overlook contradictory information, and make catastrophic decisions. Preserving human authority does not solve those problems.

Nor should human involvement become a ceremonial veto imposed on systems that may sometimes perform particular analytical tasks better than people.

The distinction is accountability.

An AI system can generate an inference. It can recommend an action. It may eventually become extraordinarily reliable at both.

But it cannot assume institutional responsibility for the consequences of acting on them.

Human agency therefore remains essential not because humans are inherently superior analytical machines, but because consequential authority requires an accountable actor capable of explaining what was known, what remained uncertain, why the available evidence justified action, and who accepted responsibility for that judgment.

The goal should not be to slow artificial intelligence until it operates at human speed.

It should be to build institutions capable of exploiting machine speed without allowing speed itself to become a substitute for judgment.

AI should help analysts see more, interrogate more possibilities, discover relationships humans might miss, and work across quantities of information that would otherwise be impossible to manage.

But somewhere between machine inference and institutional action, acceleration must encounter a boundary.

AI should accelerate what humans can know—not how quickly unverified information acquires the power to make humans act.

Notes

1. Katie Bo Lillis and Zachary Cohen, CNN, September 18, 2026. CNN reported that the intelligence product circulated earlier in the spring during the war with Iran and that military aircraft were airborne and personnel were preparing for the possibility of boarding the Chinese vessel before the operation was stopped.

2. Lillis and Cohen. CNN reported that a U.S. Special Operations Command analyst used an AI chatbot to analyze classified signals intelligence and open-source information, including vessel-manifest information; the resulting assessment incorrectly associated the cargo with a nuclear weapons program. Experienced subject-matter analysts later examined the underlying information and identified the error.

3. Ibid. CNN reported that it could not determine whether the chatbot involved was a commercial product or an internal government system and could not independently establish what the vessel was actually carrying.

4. Raymond Lopez, “Beneficial Implications of Artificial Intelligence within the United States Military: Shaping Operations,” Army Communicator (2024): 29–33.

5. C. Andrei and G. Bucăţa, “Considerations on the Impact of Artificial Intelligence on Military Engineering Operations,” International Conference KNOWLEDGE-BASED ORGANIZATION 30, no. 3 (2024): 1–7, https://doi.org/10.2478/kbo-2024-0077.

6. Lillis and Cohen, CNN, September 18, 2026.

7. U.S. Department of Defense, Responsible Artificial Intelligence Strategy and Implementation Pathway (Washington, DC: Department of Defense, June 2022). The strategy identifies Responsible, Equitable, Traceable, Reliable, and Governable as the Department’s five AI ethical principles and associates traceability with transparent and auditable methodologies, data sources, design procedures, and documentation.

8. Ibid. The strategy describes the objective as developing “justified confidence” through appropriate testing, evaluation, verification, validation, monitoring, and operator understanding of system performance and limitations.

9. M. Dawson, S. Quaye, and H. Chakraborty, “Artificial Intelligence and Irregular Warfare: Strategic Governance and Institutional Integration,” International Conference KNOWLEDGE-BASED ORGANIZATION 32, no. 3 (2026): 1–9, https://doi.org/10.2478/kbo-2026-0070.

10. Iva Atanassova, Huda Khan, and Zaheer Khan, “When Should Your Team Override AI? A Trust Calibration Routine,” Academy of Management Perspectives (2026): 1–16, https://doi.org/10.5465/amp.2025.0019. The authors distinguish automation bias from algorithm aversion and frame effective human-AI collaboration around calibrated trust rather than maximizing either reliance or skepticism.

11. Atanassova, Khan, and Khan, “When Should Your Team Override AI?” Their synthesis discusses evidence from 106 human-AI experiments and emphasizes coordination protocols governing when AI recommendations should be accepted, verified, or overridden.

12. Ibid. The authors distinguish among “AI by default,” “AI proposes, human verifies,” and “human first, AI assists” according to task risk and knowledge type.

13. Ibid. The authors identify over-deference and diffused accountability as recurring problems in human-AI teams and recommend explicit role definitions specifying what AI proposes, what humans verify, and who authorizes the final decision.

14. U.S. Department of Defense, Chief Digital and Artificial Intelligence Office, “Pathway to AI Readiness: Responsible AI,” accessed September 19, 2026.

15. M. S. Islam, “AI Application in Military Systems: Pros and Cons,” IUP Journal of Electrical & Electronics Engineering 18, no. 1 (2025): 20–43, https://doi.org/10.71329/IUPJEEE/2025.18.1.20-43.

16. Atanassova, Khan, and Khan, “When Should Your Team Override AI?” The authors argue that appropriate reliance varies with task criticality and available safeguards, with safety-critical domains warranting more conservative reliance and redundant checks.

17. U.S. Department of Defense, Responsible Artificial Intelligence Strategy and Implementation Pathway. The Department’s implementation approach applies risk-sensitive governance across the AI lifecycle while explicitly seeking to balance responsible AI practices with operational speed and ease of implementation.

18. Lillis and Cohen, CNN, September 18, 2026. CNN reported that aircraft were already airborne and personnel were preparing for the possibility of boarding the vessel before experienced analysts identified the erroneous assessment.

19. Ibid. CNN’s sources described broader concerns about AI-assisted intelligence reaching operational use without sufficient vetting, increasing AI use in targeting, and inconsistent implementation and guidance across military organizations.

20. Ibid. A source discussing the risks of accelerated AI-assisted analysis told CNN, “AI allows you to get to a bad idea faster.”

Sources

Andrei, C., and G. Bucăţa. “Considerations on the Impact of Artificial Intelligence on Military Engineering Operations.” International Conference KNOWLEDGE-BASED ORGANIZATION 30, no. 3 (2024): 1–7. https://doi.org/10.2478/kbo-2024-0077.

Atanassova, Iva, Huda Khan, and Zaheer Khan. “When Should Your Team Override AI? A Trust Calibration Routine.” Academy of Management Perspectives (2026): 1–16. https://doi.org/10.5465/amp.2025.0019. EBSCO-FullText-09_19_2026 (5).pdfPDF

Dawson, M., S. Quaye, and H. Chakraborty. “Artificial Intelligence and Irregular Warfare: Strategic Governance and Institutional Integration.” International Conference KNOWLEDGE-BASED ORGANIZATION 32, no. 3 (2026): 1–9. https://doi.org/10.2478/kbo-2026-0070.

Islam, M. S. “AI Application in Military Systems: Pros and Cons.” IUP Journal of Electrical & Electronics Engineering 18, no. 1 (2025): 20–43. https://doi.org/10.71329/IUPJEEE/2025.18.1.20-43.

Lillis, Katie Bo, and Zachary Cohen. “Exclusive: US Military Had Close Call after Using AI for False Intelligence Report, Sources Say.” CNN, September 18, 2026. https://www.cnn.com/2026/09/18/politics/us-military-ai-false-intelligence-china-ship

Lopez, Raymond. “Beneficial Implications of Artificial Intelligence within the United States Military: Shaping Operations.” Army Communicator (2024): 29–33.

U.S. Department of Defense. Responsible Artificial Intelligence Strategy and Implementation Pathway. Washington, DC: Department of Defense, June 2022.

U.S. Department of Defense, Chief Digital and Artificial Intelligence Office. “Pathway to AI Readiness: Responsible AI.” Accessed September 19, 2026.