HippoMM: Hippocampal-inspired Multimodal Memory for Long Audiovisual Event Understanding

View PDF HTML (experimental)

Abstract:Comprehending extended audiovisual experiences remains challenging for computational systems, particularly temporal integration and cross-modal associations fundamental to human episodic memory. We introduce HippoMM, a computational cognitive architecture that maps hippocampal mechanisms to solve these challenges. Rather than relying on scaling or architectural sophistication, HippoMM implements three integrated components: (i) Episodic Segmentation detects audiovisual input changes to split videos into discrete episodes, mirroring dentate gyrus pattern separation; (ii) Memory Consolidation compresses episodes into summaries with key features preserved, analogous to hippocampal memory formation; and (iii) Hierarchical Memory Retrieval first searches semantic summaries, then escalates via temporal window expansion around seed segments for cross-modal queries, mimicking CA3 pattern completion. These components jointly create an integrated system exceeding the sum of its parts. On our HippoVlog benchmark testing associative memory, HippoMM achieves state-of-the-art 78.2% accuracy while operating 5x faster than retrieval-augmented baselines. Our results demonstrate that cognitive architectures provide blueprints for next-generation multimodal understanding. The code and benchmark dataset are publicly available at this https URL.

Comments: Accepted at CVPR 2026 Findings

Subjects:

Multimedia (cs.MM); Image and Video Processing (eess.IV)

Cite as: arXiv:2504.10739 [cs.MM]

(or arXiv:2504.10739v2 [cs.MM] for this version)

https://doi.org/10.48550/arXiv.2504.10739

arXiv-issued DOI via DataCite

Submission history

From: Yueqian Lin [view email] [v1] Mon, 14 Apr 2025 22:17:55 UTC (1,822 KB) [v2] Wed, 1 Apr 2026 21:23:13 UTC (1,856 KB)