Transformer-Based Modeling of Multimodal Physiological Signals for Human Activity Recognition

Abstract

Human activity recognition has long relied on hand-crafted features and relatively shallow classifiers applied to individual sensing modalities. The rise of deep learning created new possibilities for learning directly from time-series data, but many architectures still struggle to capture long-range dependencies, multimodal interactions, and heterogeneous sampling regimes found in physiological sensing. Transformer-based models offer a promising alternative because they can attend flexibly across temporal positions and sensor channels without being constrained by strict Markov assumptions or fixed receptive fields. This paper presents a system-level analysis of transformer-based modeling for multimodal physiological signals in human activity recognition. It examines sensing infrastructure, data governance, architectural trade-offs, training strategies, deployment constraints, robustness, fairness, and sustainability. Rather than focusing on a single algorithmic contribution, the paper discusses how multimodal physiological data from wearable, optical, electrodermal, cardiac, and inertial sensors can be organized and processed through attention mechanisms. The analysis includes structural trade-offs involving centralized and distributed learning, memory and energy budgets, cross-modal fusion order, interpretability, and regulatory compliance. Special attention is given to fairness across populations, generalization under sensor variability, and the governance of health-related inference systems. The paper argues that transformer-based activity recognition should be treated not only as a modeling problem but as a socio-technical infrastructure challenge requiring careful balancing of accuracy, resource consumption, privacy, and accountability. A forward-looking perspective is offered on the deployment of these systems in clinical, industrial, and everyday contexts.