✨ Key Features
REMSA introduces several capabilities that distinguish it from existing RS retrieval systems, benchmarks, and LLM agents:
🤖 LLM-Driven Model Selection
Automates RS foundation model (RSFM) selection from natural-language queries using a modular agent workflow.
🗂️ Structured RS-FMD Database
Built on the first schema-guided database of 160+ RS foundation models with detailed metadata on modalities, sensors, pretraining, and benchmarks.
🎯 Constraint-Aware Ranking
Combines rule-based filtering with in-context LLM ranking to handle modality, sensor, resolution, compute budget, and performance requirements.
🔍 Clarification & Reasoning Loops
Asks targeted clarification questions for incomplete or ambiguous queries, improving recommendation quality.
🧠 Benchmark-Validated Performance
Evaluated on 100 real RSFM selection scenarios with 900 expert judgments, outperforming retrieval-based and RAG baselines.
Abstract
Foundation Models (FMs) are increasingly used in remote sensing (RS) for applications such as environmental monitoring, land-use mapping, and disaster response. However, selecting an appropriate RS foundation model (RSFM) is challenging due to heterogeneous documentation, diverse modality support, and complex deployment constraints. We introduce RS-FMD, the first structured and schema-guided database of over 160 RS foundation models, covering modality, sensor support, architecture, pretraining datasets, and benchmarks. Built on top of RS-FMD, we propose REMSA, the first LLM-based agent that performs automated RSFM selection from natural-language queries. REMSA interprets user intent, clarifies missing constraints, retrieves candidate models, ranks them via in-context reasoning, and generates transparent explanations. To evaluate REMSA, we construct a benchmark of 100 expert-verified remote-sensing scenarios, generating 900 scored model-query pairs using a novel rubric. Across all metrics, REMSA outperforms baselines including naive agents, dense retrieval, and unstructured RAG systems, demonstrating its utility for real RSFM selection workflows.
📊 REMSA Benchmark Coverage
| Category | Examples of Scenarios | Modalities | Typical Constraints |
|---|---|---|---|
| Land Cover Classification | Sentinel-1/2, PlanetScope, MODIS, HSI, nighttime imagery | SAR, multispectral, hyperspectral, optical | Few-shot labels, CPU-only, limited compute, OA-focused |
| Segmentation & Mapping | Flood mapping, wildfire detection, high-res urban expansion, roads | SAR, multispectral, thermal, stereo | High mIoU, fine spatial detail, real-time inference |
| Change Detection | Urban growth, coastal change, wildfire recovery | SAR, multispectral | Low compute, few-shot labels, embedded devices |
| Object Detection | Buildings, ports, industrial facilities | High-res optical | High mAP, low false positives |
| Multi-Modal Fusion | LiDAR + optical, SAR + optical, image + text | LiDAR, optical, SAR, text | High accuracy, robust fusion required |
| Self-Supervised / Unsupervised | Unlabeled SAR, HSI, LiDAR (Amazon/Arctic/Europe) | SAR, hyperspectral, LiDAR | SSL compatibility, strong representations |
| Zero-Shot / Out-of-the-Box | Flood mapping, deforestation, canopy height, HSI crops | SAR, multispectral, LiDAR, text-image | No labels available |
| Resource-Constrained Inference | Jetson Nano, Raspberry Pi, laptop-only workflows | SAR, multispectral | Lightweight models, fast inference |
📈 Quantitative Results
| System | Avg Top-1 ↑ | HQ Hit ↑ | MRR ↑ |
|---|---|---|---|
| REMSA (Ours) | 75.76 | 40.00% | 0.34 |
| REMSA-Naive | 72.67 | 37.33% | 0.29 |
| DB-Retrieval | 67.37 | 17.33% | 0.23 |
| Unstructured-RAG | 71.23 | 30.67% | 0.24 |
BibTeX
@misc{chen2026remsafoundationmodelselection,
title={REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent},
author={Binger Chen and Tacettin Emre Bök and Behnood Rasti and Volker Markl and Begüm Demir},
year={2026},
eprint={2511.17442},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2511.17442},
}