Hallucination Mitigation Using Agentic AI Natural Language-based Frameworks

Defacto Community Group,

Hello. For those interested in AI, LLMs, multi-agent systems, mitigating AI hallucinations, and fact-checking, here is an interesting new preprint:


Hallucination Mitigation Using Agentic AI Natural Language-based Frameworks
https://arxiv.org/pdf/2501.13946

"Hallucinations remain a significant challenge in current Generative AI models, undermining trust in AI systems and their reliability. This study investigates how orchestrating multiple specialized Artificial Intelligent Agents can help mitigate such hallucinations, with a focus on systems leveraging Natural Language Processing (NLP) to facilitate seamless agent interactions. To achieve this, we design a pipeline that introduces over three hundred prompts, purposefully crafted to induce hallucinations, into a front-end agent. The outputs are then systematically reviewed and refined by second- and third-level agents, each employing distinct large language models and tailored strategies to detect unverified claims, incorporate explicit disclaimers, and clarify speculative content. Additionally, we introduce a set of novel Key Performance Indicators (KPIs) specifically designed to evaluate hallucination score levels. These metrics offer a structured and quantifiable framework for assessing the impact of each agent’s refinements on the factuality and clarity of AI-generated responses. A dedicated fourth-level AI agent is employed to evaluate these KPIs, providing detailed assessments and ensuring accurate quantification of shifts in hallucination-related behaviors. A core component of this investigation is the use of the OVON (Open Voice Network) framework, which relies on universal NLP-based interfaces to transfer contextual information among agents. Through structured JSON messages, each agent communicates its assessment of the hallucination likelihood and the reasons underlying questionable content, thereby enabling the subsequent stage to refine the text without losing context. Experimental results suggest that this multi-agent, JSON-based approach not only lowers the overall hallucination scores but also renders speculative content more transparent and clearly demarcated from factual claims, improving the AI explainability level. Our findings underscore the feasibility of multi-agent orchestration and highlight the importance of maintaining a structured exchange of meta-information- particularly through formats supporting Natural Language API- to enhance the reliability and interpretability of AI-generated responses. The results demonstrate that employing multiple specialized agents capable of interoperating with each other through NLP-based agentic frameworks- such as the OVON framework- can yield promising outcomes in hallucination mitigation, ultimately bolstering trust within the AI community."

(In this preprint, Figure 1 depicts the multi-agent system orchestration structure utilized in its simulations.)

Hopefully this preprint is of some interest to the group. Perhaps, in the not-too-distant future, specialized AI agents will be able to make use of external decentralized fact-checking resources to enhance the mitigation of hallucination in multi-agent orchestrations.


Best regards,
Adam Sobieski