Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

Introduction

The success of large language models (LLMs) has triggered significant interest in enabling their visual perception capability, such that they could see, understand, and reason in the real world. At the center of these multimodal large language models (MLLMs) is a typical design where images are converted into a series of visual tokens by the vision encoders and appended with the text embeddings. CLIP is often chosen as the vision encoder since its visual representation is align with the text space by pre-training on image-text pairs. Depending on the architectures, training recipes and the way how vision tokens are injected into the language model, there exist various notable families of MLLMs such as Flamingo , BLIP , PaLI , PaLM-E and LLaVA . Most of these works keep relatively low input resolutions due to the limits on pre-trained vision encoders and LLM sequence length.

Recent studies show that stronger vision encoder design is important for mitigating MLLM hallucinations and improving resolution-sensitive tasks like optical character recognition (OCR). A constellation of works thus focuses on enhancing the capability of the vision encoder. For example, scaling up the pre-training data and parameters of vision encoder or dividing images into low-resolution patches . However, these approaches usually introduce large training resources. An efficient yet powerful strategy is to mixing visual encoders pre-trained with different tasks and input resolutions, either fusing higher resolution encoders with the CLIP encoder , sequentially appending features from different encoders , or adopting more complex fusion and routing strategies to make the best of different encoders . Such a “mixture-of-vision-experts” mode is shown to be effective. However, a detailed study of its design space with rigorous ablation is still lacking, which motivates us to revisiting this area. Questions such as which vision encoder combination to choose, how to fuse different experts together, and how to adjust training strategies with more vision encoders remain to be answered.

To address the above questions, our work systematically investigates the mixture-of-vision-encoders design space for improved MLLM perception. As shown in Fig. 1, our exploration of the design space consists of the following steps: 1) Benchmarking various vision encoders and searching recipes for higher resolution adaptation; 2) “Apples to apples” comparison between vision encoder fusion strategies; 3) Progressive identification of the optimal combination of multiple vision encoders; 4) Improved vision expert pre-alignment and data mixture. Our study covers the performance of vision encoders pre-trained on different tasks and resolutions (e.g., vision-language alignment , self-supervised learning , detection , segmentation , and OCR ). We use a round-robin approach to incorporate additional vision experts. Starting with the basic CLIP encoder, we add one additional expert at a time with the best improvement in each round.

Our work is not the first one to leverage multiple vision encoders in MLLM. However, the systematic study leads to several interesting new findings under this setting:

Unlocking the vision encoders during MLLM training matters. This is in sharp contrast to the LLaVA family and many works that consider multiple vision encoders or teachers , where freezing the vision encoders has been a common choice.

Some recently proposed fusion strategies do not show significant advantages despite their advanced designs. Instead, we find that straightforward channel concatenation stands out as a simple yet competitive fusion strategy, offering the best efficiency and performance.

Incorporating additional vision experts leads to consistent gain, making it a promising path to systematic enhance MLLM perception besides scaling up single encoders. The improvement is particularly pronounced when vision encoders are unlocked.

We propose a pre-alignment stage where non-text-aligned vision experts are individually fine-tuned with a frozen LLM before trained together. This stage is found to enhance the MLLM performance significantly under the mixture-of-vision-encoder design.

We finally conclude our findings into a family of MLLMs termed Eagle. Eagle is evaluated on a series of benchmarks, including visual question answering, OCR/document related tasks, and benchmarks tailored for MLLMs. Our model attains state-of-the-art performance across different benchmarks and demonstrates obvious advantages on OCR and document understanding tasks. Using the same pre-train and supervised fine-tuning data from Cambrian-1 - a concurrent family of vision-centric MLLMs sharing similar design spirits, Eagle models overall achieve better performance. We hope that the Eagle can provide a highly performant and easy-to-reproduce MLLM solution to the community.

Design space exploration

In this section, we show how to utilize the advantages of different vision encoders via step-by-step investigations, yielding the Eagle model family. Unlike previous methods focusing on new fusion strategies or architectures among vision encodes, our goal is to identify a set of minimalistic design to fuse different vision encoders supported with detailed ablations, removing any unnecessary parts. As shown in Fig. 2, we start from extending the basic CLIP encoder to a set of vision experts with different architectures, pre-training tasks and resolutions. With these experts, we then compare different fusion architectures and methods, and study how to optimize the pre-training stratgies given more encoders. We also give a detailed analysis in how to select the vision encoders to be integrated. Finally, we put all the findings together and further extend to multiple expert vision encoders with different resolutions and domain knowledge.

We adopt LLaVA’s model architecture as the basis, which consists of a large language model, a vision encoder, and a projection layer. The projection layer projects the visual embedding from the vision encoder into the text embedding space. We mainly focus on the vision encoder design.

Base training data. We use the same pre-training data as LLaVA-$1.5$ , which consists of $595$k image text pairs. For the supervised fine-tuning stage, we collect data from a series of tasks and convert them into multimodal conversations, including: LLaVA-1.5 , Laion-GPT4V , ShareGPT-4V , DocVQA , synDog-EN , ChartQA , DVQA , and AI2D , resulting in 934k samples.

Implementation details. We first pre-train the model with image-text pairs for one epoch with a batch size of 256, where the whole model is frozen and only the projector layer is updated. In the second stage, we further fine-tune the model on the supervised fine-tuning data for one epoch with a batch size of 128. For this exploration, we employ Vicuna-7B as the underlying language model. The learning rates are set to 1e-3 for the first stage and 2e-5 for the second stage, respectively.

2 Stronger CLIP encoder

We start our exploration with the CLIP model since it has become the main choice for many MLLMs. While CLIP models are known to benefit multimodal tasks, their drawbacks have also been well-studied. For instance, many existing MLLMs tend to use the pre-trained CLIP resolutions (such as $224\times 224$ or $336\times 336$) as their input resolutions. In these cases, the encoders often fail to capture fine-grained details that are important for resolution-sensitive tasks like OCR and document understanding .

To handle increased input resolution, a common practice is to use tiling where input images are divided into tiles and encoded separately . Another simpler method is to directly scale up the input resolution and interpolate the position embeddings of the vision transformer model if needed . We compare these two approaches with frozen/unfrozen vision encoders under different resolutions, with the results shown in Table 1. Our findings can be summarized as follows:

Unfreezing the CLIP encoder leads to significant improvement when interpolating to a higher MLLM input resolution that is different from the CLIP pre-training resolution. There is also no performance degradation when resolutions remain the same.

When the CLIP encoder is frozen, directly adapting it to a higher MLLM input resolution considerably hurts the performance.

Among the compared strategies, directly interpolating to $448\times 448$ with an unfrozen CLIP encoder is shown to be both effective and efficient in terms of performance and cost.

The best CLIP encoder gets close to InternVL in performance despite the significantly smaller model size (300M vs. 6B) and less pre-training data.

It is worth noting that CLIP-448 allows us to match the setting with LLaVA-HR and InternVL where the CLIP encoders are similarly adapted to take $448\times 448$ input and output $1024$ patch tokensThe latest InternVL 1.0 and InternVL 1.2 models also adopt a pixel shuffle module. We remove it in our experiment to ensure full alignment. See https://github.com/OpenGVLab/InternVL for details.. In the following investigation, we stick to this simple strategy of scaling up the input resolution and unlocking the vision encoder during training.

3 Vision experts

To better establish the foundation for multi-vision expert fusion, we extend the toolbox with vision experts pre-trained on different tasks and resolutions, and verify our findings on high-resolution adaptation with these experts. This also helps us identify the distinct advantages of different experts. We collect a set of vision experts, including: (1) Vision-Language Alignment: CLIP and ConvNeXt from OpenCLIP . (2) Object-Centric: EVA-02 pre-trained on detection datasets. (3) OCR: Pix2Struct .(4) Segmentation: SAM . (5) Self-supervised: DINOv2 . The detailed input resolution and checkpoint of each vision encoder can be found in Table 2. We resize the output 2D feature maps of each vision encoder using bilinear interpolation and pixel shuffle to ensure that the visual token number equals $1024$.

The results in Table 3 reveal that MLLMs with these task-specific vision encoders achieve optimal performance in their pre-training domains. EVA-02 excels in the object hallucination evaluation benchmark POPE and general visual question answering benchmark GQA . CLIP and ConvNeXt perform well across all benchmarks, benefiting from their training on large-scale image-text pairs using contrastive loss. Conversely, while Pix2Struct excels in text recognition, it shows limited capability in object recognition and general VQA tasks, like POPE and GQA. DINOv2 and SAM, pre-trained with self-supervised learning and semantic segmentation respectively, struggle with text recognition tasks. Note that the conclusion made in Sec. 2.2 aligns with the results here. Both ConvNeXt and EVA-02 show significant performance gains when the vision encoders are updated during training.

4 Fusion strategy

Existing MLLM frameworks have proposed various fusion strategies to ensemble the encoders in Table 3, with the hope that their domain-specific strengths can be leveraged. In all cases, improvements in MLLM performance have been reported with the fusion of vision encoders. However, the roles of the fusion strategies as part of their MLLM architecture innovations, have not been decoupled and clearly studied under an “apples to apples” comparison. It is thus not entirely clear how much improvement is from the fusion strategies themselves versus the improved representations from various encoders.

We notice that existing popular fusion strategies, despite their variations in designs, can be broadly represented by the following several categories: (1) Sequence Append: directly appending the visual tokens from different backbones as a longer sequence ; (2) Channel Concatenation: concatenating the visual tokens along the channel dimension without increasing the sequence length ; (3) LLaVA-HR: injecting high-resolution features into low-resolution vision encoders using mixture-of-resolution adapter ; (4) Mini-Gemini: using the CLIP tokens as the low resolution queries to cross-attend another high-resolution vision encoder in the co-located local windows . (5) Deformable Attention: a new baseline we introduce on top of Mini-Gemini, where the vanilla window attention is replaced with deformable attention . Fig. 2 gives a detailed illustration of these fusion strategies. To better study them, we choose “CLIP+ConvNeXt” and “CLIP+ConvNeXt+SAM” as the base multi-encoder combinations to perform comparisons.

Our study in Table 4 shows that Channel Concatenation achieves the best average performance while maintaining better throughput compared to sequence append. The “injection-based” methods, such as LLaVA-HR, Mini-Gemini and Deformable Attention, are in general less competitive on TextVQA and OCRBench , performing worse than using ConvNeXt alone as the vision encoder. A plausible explanation is that the CLIP features continue to play a dominant role in the visual tokens. Although sequence append shows comparable performance to channel concatenation, it faces the challenge to handle more vision encoders due to the increasing sequence length. Hence, we choose direct channel concatenation as our fusion strategy considering its performance, expandability, and efficiency.

5 Vison-language Pre-Alignment

As shown in Table 3, vision backbones pre-trained exclusively on vision tasks (e.g., detection, OCR, and segmentation) are less competitive compared to encoder pre-trained on vision language alignment. This is possibly due to representational inconsistencies when integrated with large language models. Additionally, when combining different encoders, there is a gap between these encoders, creating difficulties in the training process. To address this feature inconsistency, we propose a Pre-Alignment training stage that first aligns each individual vision encoder with the same large language model, fostering better synergy between visual and linguistic capabilities.

Fig. 3 depicts our pre-alignment strategy. Instead of training a projector to simultaneously align multiple vision experts as in LLaVA’s original pre-training strategy, we first align the representation of each individual expert with a smaller language model (Vicuna-7B in practice) using next-token-prediction supervision. As shown in Fig. 3, with pre-alignment, the whole training process consists of three steps: 1) training each pre-trained vision expert with their own projector on SFT data, while keeping the language model frozen; 2) combining all the vision experts from the first step and training only the projector with image-text pairs data; 3) training the whole model on the SFT data.

To verify the proposed method, we compare the pre-alignment strategy with the normal two-stage training strategy in Table 5, considering both freezing and unfreezing vision experts for comparison. As shown in Table 5, although unfreezing the vision experts during SFT helps improve performance by updating the vision experts to fit the language model, the Pre-Align strategy more effectively mitigates the inherent biases of each vision expert and stabilizes the training process, subsequently improving overall performance.

6 Extension to multi-experts

With the optimized strategies and training recipes of incorporating individual vision experts, we consider the incorporation of even more vision experts to push the limit. To conduct the searching in a systematic and principled manner, we adopt a step-by-step greedy strategy to incorporate additional vision experts.

We consider the vision experts discussed in Section 2.3 for experiments. We mark CLIP, ConvNeXt, SAM, DINOv2, Pix2Struct, and EVA-02-L as A, B, C, D, E, and F, respectively. A round-robin scheme, as shown in Table 6, is adopted. We first use the two top performing vision encoders, CLIP and ConvNeXt, as the basis and gradually add one more vision encoder each time. In each round, the best-performing vision encoder combination is retained for the next round.

Table 6 reveals several insights. Generally, introducing additional vision encoders enhances the performance. This indicates that the distinct advantages of different encoders can be preserved and utilized; for example, integrating the EVA-02 encoder improves metrics on the POPE benchmark. Although individual metrics may vary, the aggregated performance shows an upward trend, as evidenced by normalized average metrics, suggesting that the overall efficacy of the system is enhanced with more encoders. Also, Table 6 shows that the best combination of vision experts are CLIP, ConvNeXt, SAM, Pix2Struct, and EVA-02. We will use this recipe in our final model.

Experiments

After meticulously developing our recipes, we have formulated the following principals for our model, Eagle: (1) integrating more vision experts with optimized training recipe; (2) combining multiple vision experts through direct channel concatenation; (3) pre-training the vision experts separately via pre-alignment. In this section, to further show the advantages of the Eagle models, we add more training data and compare Eagle against the current state-of-the-art MLLMs on different tasks.

Models. We use Vicuna-v1.5-7B , Llama3-8B and Vicuna-v1.5-13B as the language model. For vision encoder, following the results in Sec. 2.6, we denote the model with four vision encoders: CLIP, ConvNeXt, Pix2Struct and EVA-02 as Eagle-X4 and the model with an additional SAM vision encoder as Eagle-X5.

Augmented training data. Considering the training cost, we only use 934K data during the investigation and ablations, as listed in Sec. 2.1. In this section, inspired by recent practices , we introduce more datasets, including LVIS-INSTRUCT4V , LRV , Geo170K , LLaVAR , Visual7w , and OpenHermes-2.5 , resulting in a total of 1,809k data for the supervised fine-tuning stage. See Table 7 for more details on this data mixture. To better perform an apple-to-apple comparison with recent open-source MLLM projects, we also train Eagle-X5 with the same data as Cambrian-1 , including 2.5 million and 7 million data for pre-training and supervised fine-tuning.

2 Main results

Evaluation on visual question answering tasks. We compare Eagle model series across three Visual Question Answering (VQA) benchmarks, including GQA , VQAv2 and VizWiz . As shown in Table 8, Eagle-X5 achieves state-of-the-art performance on GQA and VQAv2, underscoring the advantages of incorporating additional vision experts.

Evaluation on OCR and chart understanding tasks. To evaluate the OCR, document and chart understanding capabilities of Eagle, we benchmark our model on OCRBench , TextVQA , and ChartQA . As illustrated in Table 8, our model significantly surpasses competitors on TextVQA, benefiting from its high-resolution architecture and integration of different vision encoders. Notably, Eagle maintains a straightforward design, supporting up to 1024 tokens without requiring complex tile decomposition of images.

Fig. 5 shows some examples of OCR and document understanding cases. With high-resolution adaptation and more vision experts, our model can identifies small text within images and accurately extracts information according to the users’ instructions. To better understand the benefits of introducing experts pre-trained on other vision tasks,we visualize the results of a model with only the ConvNeXt and CLIP vision encoders, compared to the results of Eagle-X5 in Fig. 4. With the full set of vision encoders, the model can successfully correct mistakes, showing that even when equipped with high-resolution vision encoders pre-trained on vision-language alignment, the model’s abilities can still be enhanced by integrating additional vision experts pre-trained on diverse vision tasks.

Evaluation on multimodal benchmarks. We evaluate Eagle on seven benchmarks for MLLMs to demonstrate its capabilities from different perspectives, including MME , MMBench , SEED , MathVista , MMMU , ScienceQA , and POPE . Specifically, MME, MMBench, and SEED assess the overall performance on various real-world tasks based on reasoning, recognition, knowledge, and OCR. MMMU focuses on challenging problems from diverse domains that require college-level knowledge. POPE evaluates the visual hallucinations of MLLMs. The metrics used in our paper adhere to the default settings of these benchmarks. We report the perception score for MME, the en_dev split for MMBench, the image split of SEED, the test-mini split of MathVista, the val split of MMMU, the F1-score of POPE, and the image of SQA to align with the reported scores from other models.

From the data presented in Table 8, Eagle consistently surpasses existing models across various MLLMs on SEED and MME, demonstrating the comprehensive knowledge and reasoning abilities of Eagle. With the help of vision encoders on object-centric tasks, Eagle also achieves the best performance on the POPE benchmark. Additionally, the Pre-Alignment strategy discussed in Sec. 2.5 has been found to further enhance performance when integrating multiple task-specific vision backbones. This approach not only mitigates the inherent biases of each vision expert and the synergy between different modalities but also establishes a robust framework for the multiple-expert fusion.

Comparison with Cambrian-1. Using the same pre-training and supervised fine-tuning datasets from Cambrian-1 , Eagle demonstrates superior performance across all the evaluated benchmarks without bells and whistles. As shown in Table 9, Eagle outperforms the Cambrian-1 counterparts considerably on the OCR and Chart category. Consistent improvements are also observed on the General, Knowledge and Vision-Centric categories, showing the robustness and generalization ability of the improved perception design in Eagle.

Related work

Our work is related to the general architecture design of multimodal large language models. Besides the line of representative open-source research mentioned in the introduction section, other notable families of MLLMs include, but are not limited to MiniGPT-4 , Lynx , Otter , Qwen-VL , CogVLM , VILA , GPT-4V , Gemini , and Llama 3.1 . Depending on how vision signals are integrated into the language model, MLLMs can be broadly categorized into “cross-modal attention” ones and “prefix-tuning” ones . The former injects visual information into different layers of LLMs using cross-modal attention , whereas the latter views the visual tokens as part of the language token sequence and directly append them with text embeddings . Our model belongs to the prefix-tuning family by following a LLaVA-styled multimodal architecture. Considering that MLLM is a fast growing field, readers are recommended to refer to more detailed studies and surveys such as .

2 Vision encoder design for MLLMs

Our work is closely related to research focusing on improved vision encoder designs for MLLM. Early works usually adopted vision encoder pre-trained on vision-language alignment tasks such as CLIP and EVA-CLIP . Stronger vision encoders such as SigLIP and InternVL have been proposed to improve vision-language tasks with improved designs, larger model sizes, and better training recipes. Considering that the models are often pre-trained on low-resolution images and lack the ability to encode fine-grained details, higher resolution adaptation is often performed to increase the MLLM input resolution .

Besides higher resolution adaptation, models such as LLaVA-NeXT , LLaVA-UHD , Monkey , InternLM-XComposer , and InternVL use either tiling or adaptive tiling to handle high resolution input, where input images are divided into lower-resolution patches and process them separately. Although the ability to handle higher resolution is similarly made possible with the introduction of additional vision experts, its spirit is slightly orthogonal to the tiling techniques in terms of introducing the diversity of visual representations. Both techniques are compatible and can be combined together.

Our work is most related to existing models using multiple vision encoders for improved perception. Mini-Gemini and LLaVA-HR propose to fuse high-resolution visual features into the low-resolution visual tokens. Apart from the resolution issue, these pre-trained vision encoders may lack specific abilities such as reading text and localizing objects. Hence, a series of works have integrated vision models pre-trained on various vision tasks to extend the capabilities of vision encoders. For example, Mousi , and Brave fuse visual tokens from different vision encoders by concatenating along the channel or token direction. RADIO proposes a multi-teacher distillation method to unify the abilities of different vision encoders into a unified model. MoAI , IVE , and Prismer further use the output of vision experts, such as OCR, detection, or depth estimation, to supplement more information for MLLM to generate answers. MoVA devises a routing network to assign an optimal vision model conditioned on the given image and instructions.

Conclusion

We conduct an in-depth analysis study on the design space for integrating vision encoders for multimodal large language models. Unlike previous works that focus on designing novel fusion paradigms, we find systematic design choice matters and discover a series of useful techniques. Step by step, we optimize the training recipe of individual vision encoders, identify an extendable and efficient fusing method, and gradually combine vision encoders with different domain knowledge. The results show the importance of basic design space. We hope our work can serve as a new basis and bring new inspiration for the vision encoder design for MLLM.

References