Towards Knowledge Guided Pretraining Approaches for Multimodal Foundation Models: Applications in Remote Sensing

View PDF HTML (experimental)

Abstract:Self-supervised learning has emerged as a powerful paradigm for pretraining foundation models using large-scale data. Existing pretraining approaches predominantly rely on masked reconstruction or next-token prediction strategies, demonstrating strong performance across various downstream tasks, including geoscience applications. However, these approaches do not fully capture the knowledge of causal interplay between different geospatial and environmental variables. To address this limitation, we propose Knowledge Guided Variable-Step Forecasting (KG-VSF), a novel pretraining task that models forecasting as a conditional generation task, where driver variables (e.g., weather) inform the prediction of response variables (e.g., satellite imagery). We demonstrate that pretraining in such a fashion leads to strong embeddings which give enhanced performance when finetuned on downstream tasks where capturing this causality matters such as pixel wise crop type mapping, soil moisture estimation and forecasting, missing image prediction, and future image forecasting when compared to finetuning embeddings from other standard pretraining approaches.

Comments: 33 pages with appendix

Subjects:

Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)

Cite as: arXiv:2407.19660 [cs.CV]

(or arXiv:2407.19660v5 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2407.19660

arXiv-issued DOI via DataCite

Submission history

From: Praveen Ravirathinam [view email] [v1] Mon, 29 Jul 2024 02:49:55 UTC (1,399 KB) [v2] Wed, 16 Oct 2024 21:18:10 UTC (3,245 KB) [v3] Tue, 18 Feb 2025 03:39:37 UTC (15,914 KB) [v4] Fri, 2 Jan 2026 09:15:07 UTC (4,709 KB) [v5] Fri, 27 Mar 2026 01:24:51 UTC (4,735 KB)