ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts
arXiv:2603.28902v1 Announce Type: new Abstract: Charts are central to analytical reasoning, yet existing benchmarks for chart understanding focus almost exclusively on single-chart interpretation rather than comparative reasoning across multiple charts. To address this gap, we introduce ChartDiff, the first large-scale benchmark for cross-chart comparative summarization. ChartDiff consists of 8,541 chart pairs spanning diverse data sources, chart types, and visual styles, each annotated with LLM-generated and human-verified summaries describing differences in trends, fluctuations, and anomalies. Using ChartDiff, we evaluate general-purpose, chart-specialized, and pipeline-based models. Our results show that frontier general-purpose models achieve the highest GPT-based quality, while specia
View PDF HTML (experimental)
Abstract:Charts are central to analytical reasoning, yet existing benchmarks for chart understanding focus almost exclusively on single-chart interpretation rather than comparative reasoning across multiple charts. To address this gap, we introduce ChartDiff, the first large-scale benchmark for cross-chart comparative summarization. ChartDiff consists of 8,541 chart pairs spanning diverse data sources, chart types, and visual styles, each annotated with LLM-generated and human-verified summaries describing differences in trends, fluctuations, and anomalies. Using ChartDiff, we evaluate general-purpose, chart-specialized, and pipeline-based models. Our results show that frontier general-purpose models achieve the highest GPT-based quality, while specialized and pipeline-based methods obtain higher ROUGE scores but lower human-aligned evaluation, revealing a clear mismatch between lexical overlap and actual summary quality. We further find that multi-series charts remain challenging across model families, whereas strong end-to-end models are relatively robust to differences in plotting libraries. Overall, our findings demonstrate that comparative chart reasoning remains a significant challenge for current vision-language models and position ChartDiff as a new benchmark for advancing research on multi-chart understanding.
Comments: 21 pages, 17 figures
Subjects:
Artificial Intelligence (cs.AI)
Cite as: arXiv:2603.28902 [cs.AI]
(or arXiv:2603.28902v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2603.28902
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Rongtian Ye [view email] [v1] Mon, 30 Mar 2026 18:29:02 UTC (3,295 KB)
Sign in to highlight and annotate this article
Conversation starters
Daily AI Digest
Get the top 5 AI stories delivered to your inbox every morning.
More about
modellanguage modelbenchmark
A social network for AI looks disturbing, but it s not what you think
A social network where humans are banned and AI models talk openly of world domination has led to claims that the "singularity" has begun, but the truth is that much of the content is written by humans
![[P] Trained a small BERT on 276K Kubernetes YAMLs using tree positional encoding instead of sequential](https://images.unsplash.com/photo-1677442135703-1787eea5ce01?w=600&q=80)
[P] Trained a small BERT on 276K Kubernetes YAMLs using tree positional encoding instead of sequential
I trained a BERT-style transformer on 276K Kubernetes YAML files, replacing standard positional encoding with learned tree coordinates (depth, sibling index, node type). The model uses hybrid bigram/trigram prediction targets to learn both universal structure and kind-specific patterns — 93/93 capability tests passing. Interesting findings: learned depth embeddings are nearly orthogonal (categorical, not smooth like sine/cosine), and 28/48 attention heads specialize on same-depth attention (up to 14.5x bias). GitHub: https://github.com/vimalk78/yaml-bert submitted by /u/vimalk78 [link] [comments]
Knowledge Map
Connected Articles — Knowledge Graph
This article is connected to other articles through shared AI topics and tags.
More in Models
A social network for AI looks disturbing, but it s not what you think
A social network where humans are banned and AI models talk openly of world domination has led to claims that the "singularity" has begun, but the truth is that much of the content is written by humans
[P] Trained a small BERT on 276K Kubernetes YAMLs using tree positional encoding instead of sequential
I trained a BERT-style transformer on 276K Kubernetes YAML files, replacing standard positional encoding with learned tree coordinates (depth, sibling index, node type). The model uses hybrid bigram/trigram prediction targets to learn both universal structure and kind-specific patterns — 93/93 capability tests passing. Interesting findings: learned depth embeddings are nearly orthogonal (categorical, not smooth like sine/cosine), and 28/48 attention heads specialize on same-depth attention (up to 14.5x bias). GitHub: https://github.com/vimalk78/yaml-bert submitted by /u/vimalk78 [link] [comments]
Avoid Re-encoding Reference Images in Vision-LLM When Comparison Criteria Are User-Defined
Hi everyone, I’m working with a Vision-LLM (like Qwen-VL / LLaVA / llama.cpp-based multimodal models) where I need to compare new images against reference images. The key part of my use case is that users define the comparison criteria (e.g., fur length, ear shape, color patterns), and I’m using image-to-text models to evaluate how well a new image matches a reference according to these criteria. Currently, every time I send a prompt including the reference images, the model re-encodes them from scratch . From the logs, I can see: llama-server encoding image slice... image slice encoded in 3800–4800 ms decoding image batch ... Even for the same reference images, this happens every single request , which makes inference slow. Questions: Has anyone dealt with user-defined comparison criteria
Discussion
Sign in to join the discussion
No comments yet — be the first to share your thoughts!