Parameter-Efficient Fine-Tuning for Hallucination-Aware Multimodal Models

Abstract

Multimodal models increasingly combine visual, textual, auditory, and structured data to support decision-making in domains such as healthcare, mobility, industrial inspection, education, and public administration. Their capacity to generate fluent cross-modal descriptions and recommendations, however, is accompanied by hallucination risks that can undermine trust, safety, and regulatory compliance. This paper examines parameter-efficient fine-tuning as a system-level strategy for hallucination-aware multimodal models. Rather than treating adaptation as a narrow optimization problem, the analysis frames it as an architectural, infrastructural, and governance concern. It reviews adapter-based, low-rank, prompt-based, and quantization-aware approaches, then connects them to multimodal hallucination sources such as weak grounding, attention misallocation, dataset bias, and evaluation gaps. The discussion emphasizes structural trade-offs involving memory, latency, modularity, observability, fairness, sustainability, and policy alignment. It argues that parameter-efficient fine-tuning can reduce computational barriers and enable domain-specific correction, but it can also concentrate risk when hallucination-aware objectives are absent from training data, evaluation pipelines, and deployment controls. The paper concludes that robust multimodal systems require layered governance, continuous auditing, modality-aware benchmarks, and infrastructure that makes adaptation reversible, traceable, and accountable. Such an approach positions parameter-efficient fine-tuning not as a shortcut for model customization, but as a component of socio-technical assurance for generative multimodal infrastructures.