Composition-Grounded Instruction Synthesis for Visual Reasoning

COGS operates in 3 stages:

1) Seed Data Decomposition: given a small seed set of complex questions, a multimodal LLM decomposes each question into interpretable perception and reasoning factors. The aggregated factor set captures the seed domain’s compositional structure.

2) Question Generation via Factor Recomposition: given a new image and sampled factors from decomposed set, a multimodal LLM generates grounded subquestions, composes them into a complex question, and outputs both intermediate answer of subquestions and overall answers, enabling annotation-free data expansion;

3) Reinforcement Learning-Based Fine-tuning: we adopt GRPO to fine-tune a pretrained MLLM with the generated question–answer data. The structure of data generated COGS enables richer reward modeling beyond final-answer correctness.

Chart Question Answering (CQA) requires interpreting visual representations in charts and reasoning over their spatial relation and underlying data. The recently released ChartQAPro benchmark consists of 1,948 human-curated question–answer pairs targeting complex reasoning over diverse chart types.

we randomly select 33% of the released test set as validation data and treat them as seed questions for data synthesis. The remaining 67% is held out as a fully unseen test set for all experiments. we use the training set of ChartQA as the image source.

To demonstrate the generality of COGS, we also evaluate it on the webpage question answering domain, which requires visual, semantic, and structural reasoning over graphical user interfaces (GUIs). We adopt VisualWebBench, a benchmark consisting of diverse real-world webpages paired with reasoning-intensive, human-curated questions.

We use questions from VisualWebBench as seeds and screenshots from MultiUI as the image source.

We extend COGS to a multi-dataset setting by incorporating the MultiModal Chart Benchmark (MMC-Bench).

We compare two strategies for synthesizing data across domains:

1. Data-level mixture: decompose and recompose A and B independently, then combine the synthesized data, i.e., Recompose(Decompose(A)) + Recompose(Decompose(B)).

2. Factor-level mixture: decompose A and B separately, merge all extracted factors into a joint pool, and recompose using this combined pool, i.e., Recompose(Decompose(A) ∪ Decompose(B)).

In addition, we include two "specialist models" trained only with augmented data from a single domain (e.g., trained on augmented A and evaluated on A). These serve as "upper-bound references" for in-domain data augmentation. All methods use Qwen2.5-VL-7B as the base model and are trained with GRPO and ProcessRM-max.

The results show that factor-level mixture is a better strategy for data mixing.

We ablate 3 reward models.

• StandardRM: r(y) = r^final(y), which only evaluates final-answer correctness. This is the default option when subquestion supervision is not available.
• ProcessRM-sum: r(y) = r^final(y) + λ ⋅ r^sub(y), which combines correctness of the final answer with the average subquestion accuracy, encouraging faithful reasoning at the factor level.
• ProcessRM-max: r(y) = max(r^final(y), λ ⋅ r^sub(y)), which prioritizes the final answer but still provides reward shaping when the intermediate reasoning is largely correct.

Theoretical proof* and empirical results show that ProcessRM-max is the most effective reward model in the setting of COGS.

*Please find more details in our paper.