. LLM은 셀 수 없습니다: GPT, Gemini 및 Claude에 대한 환각 분류

Quantitative Analysis of Hallucination Bias in LLM Counting Tasks and Suppression Effects via Structured Protocol (KIS) Description Abstract (English) This study presents an exploratory quantitative analysis of hallucinations arising when large language models (LLMs) count items in large volumes of unstructured text data, and examines the suppression effects of the Knowledge Innovation System (KIS), a proprietary structured protocol. Three models — GPT-5.3 Instant, Gemini 3 Flash, and Claude Sonnet 4.6 — were evaluated on a three-label (Yes / No / Pending) text dataset ranging from 200 to 2,000 items under four conditions: Baseline (no protocol), KIS Level 4 / Logic: Strict, Chain-of-Thought (CoT) prompting, and a KIS + CoT hybrid. Results showed that Gemini overcounted the Pending category by +38 items at 1,000 entries under Baseline, exhibiting what we term harmonic hallucination, yet achieved 100% accuracy across all scales with KIS applied. Claude maintained perfect accuracy up to 2,000 items without any protocol. ChatGPT abandoned the task beyond 800 items under Baseline but recovered to 100% accuracy at 1,000 items under the KIS + CoT hybrid. Notably, applying CoT alone to ChatGPT induced distribution fabrication even at 200 items, demonstrating a counter-productive effect. Based on these findings, we propose a three-type taxonomy of LLM hallucination: Confabulation Type (Gemini), Avoidance Type (ChatGPT), and Process-Opaque Type (Claude). We further demonstrate that KIS functions as an external scaffold — structurally separating the counting, verification, and reporting phases via its log: full output — thereby leveling inter-model performance gaps and providing the audit trails required in practical deployments. Keywords: LLM, Hallucination, Counting Task, Prompt Engineering, KIS, Chain-of-Thought, Model Comparison, Audit Trail Abstract (Japanese) Abstract(日本語) 本研究は,大規模言語モデル(LLM)が大量の非構造テキストデータを計数する際に生じるハルシネーション(統計的分布の捏造)を定量的に分析し,独自の構造化プロトコルであるKnowledge Innovation System(KIS)による抑制効果を検証した探索的研究である. 3モデル(GPT-5.3 Instant,Gemini 3 Flash,Claude Sonnet 4.6)を対象に,200〜2,000件の「はい/いいえ/保留」形式テキストデータを用い,KIS無し(Baseline),KIS有り(Level 4 / Logic: Strict),外形プロンプト(CoT),KIS+外形プロンプトの4条件で比較実験を行った. 実験の結果,Geminiは1,000件のBaseline条件で「保留」を+38件過剰計数する「調和的ハルシネーション」を示したが,KIS適用により全水準で精度100%を達成した.Claudeは素の状態で2,000件まで完全精度を維持した.ChatGPTは800件以上でタスクを放棄したが,KIS+外形プロンプトの組み合わせにより1,000件で精度100%に回帰した.また,ChatGPTへの外形プロンプト単体の適用は200件時点で分布捏造を誘発し,逆効果であることが確認された. これらの結果から,LLMのハルシネーションは「捏造型(Gemini)」「回避型(ChatGPT)」「プロセス隠蔽型(Claude)」の3類型に分類できることを提案する.さらに,KISのlog: full機能が計数・検証・報告のフェーズを構造的に分離する「外骨格(external scaffold)」として機能し,モデル間の性能差を平準化するとともに,実務に求められる監査証跡(audit trail)を提供することを示す. キーワード:大規模言語モデル,ハルシネーション,計数タスク,プロンプトエンジニアリング,KIS,Chain-of-Thought,モデル比較,監査証跡 Files LLM_Counting_Hallucination_EN.pdf Additional details Additional titles - Translated title (Japanese) - LLMの計数タスクにおける ハルシネーションの定量分析と 構造化プロトコル(KIS)による抑制効果の検証 - [1] Wei et al. (2022) Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 - [2] Huang et al. (2023/2024) A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv:2311.05232. ACM TOIS. - [3] Anh-Hoang et al. (2025) Survey and analysis of hallucinations in large language models: attribution to prompting strategies or model behavior. Frontiers in AI, 8:1622292. - [4] Meincke, Mollick et al. (2025) Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting. Wharton School Research Paper. SSRN:5285532. - [5] Zhao et al. / NumericBench (2025) Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models. arXiv:2502.11075. - [6] Chandra et al. (2025) Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians. arXiv:2602.19141. - [7] Hasegawa & Kamogawa (2026a) KIS: A Question-Centric Protocol Architecture for Hierarchical AI Thought Control. Zenodo. DOI:10.5281/zenodo.18730671. Published: 2026-02-22. - [8] Hasegawa & Kamogawa (2026b) KIS-Genesis v4.2: A Question-Centric Protocol Architecture for Hierarchical AI Thought Control. Zenodo. DOI:10.5281/zenodo.18951932. Published: 2026-03-11.