Return to page
h2o_logo h2o_logo
Request Live Demo
On-demand Demos
X

H2OVL Mississippi

The first-ever multimodal H2O model

H2OVL Mississippi-2B and 0.8B are two powerful new multimodal foundation models designed specifically for OCR and Document AI use cases. Released under the Apache 2.0.

Now available on Hugging Face

H2OVL Mississippi-0.8BH2OVL Mississippi-2B

An image depicting H2OVL Mississippi actively crawling and referencing a PDF. The PDF itself is a PDF of an invoice for property services given from HULSEY propert lawyers, P.C. An image depicting H2OVL Mississippi actively crawling and referencing a PDF. The PDF itself is a PDF of an invoice for property services given from HULSEY propert lawyers, P.C.

#1 SLM for Text Recognition

H2OVL Mississippi 0.8B Model surpasses leading Small Vision Language Models (SVLMs)

Impressively outperforms even larger state-of-the-art Vision Language Models (VLMs) like InternVL2-26B that are 25x bigger than our model, delivering economic efficiency and ease of deployment for text recognition applications.

A chart showing the text recognition performance of various vision language models per the OCRBench benchmark. Mississippi is top performer. A chart showing the text recognition performance of various vision language models per the OCRBench benchmark. Mississippi is top performer.
A scatterplot graph depicting various vision language models performance on text recognition relative to their model size. Mississippi is the smallest and highest performing model. A scatterplot graph depicting various vision language models performance on text recognition relative to their model size. Mississippi is the smallest and highest performing model.

H2OVL Mississippi 0.8B Model surpassed all comparable SLMs in the market on OCR benchmarks

Beats all leading SLMs including Microsoft Phi-3 Vision and Google PaliGemma-3B-mix-448 on Text Recognition in OCRBench.

Built on the Danube3-0.5B, H2OVL Mississippi-0.8B model-pre-trained on 11 million conversation pairs and fine-tuned with an additional 8 million, surpassed all comparable SLMs on OCR benchmarks.

See it on Hugging Face

H2OVL Mississippi 2B

H2OVL Mississippi 2B is built on H2O Danube2 with a robust 2.1 billion parameter model optimized for lightweight deployment. This specialized multimodal architecture blends language and computer vision to meet the growing demand for more economical multimodal OCR.

H2OVL Mississippi 2B is pre-trained on 5.3M conversation pairs and fine-tuned with an additional 12M pairs using advanced image processing techniques to handle high-resolution images.

H2OVL Mississippi 2B rivals state-of-the-art SLMs on single-image benchmarks.

A chart depicting Image Benchmarks across various computer vision models. Mississippi is #2 A chart depicting Image Benchmarks across various computer vision models. Mississippi is #2

Model architecture

A diagram depicting the H2OVL Mississippi model architecture a-diagram-depicting-the-h2ovl-mississippi-model-architecture

Mississippi Model Architecture: This diagram shows how input images and text are processed. Images are resized and cropped in three ways: (a) closest original size, (b) different aspect ratio, and (c) thumbnail.

The architecture, inspired by LLaVA and InternVL, uses a ViT-MLP-LLM setup with a vision encoder (InternViT-300M) and a language model (Danube-2 or Danube-3) for varying computational needs.

Dynamic resolution divides images into 448x448 pixel tiles (up to 4K) based on aspect ratio, similar to GPT-4V's modes, balancing detail and efficiency.

H2OVL Mississippi-2B also uses multi-scale adaptive cropping (MSAC) to reduce sawtooth effects and enhance feature capture for tasks like document parsing and image recognition.

Training methodology

A two-stage training approach was used for Mississippi models to align visual features with the language model. One-stage training often fails to achieve this alignment in smaller models.

Pretraining

For H2OVL Mississippi-2B, the vision encoder (InternViT-300M) and MLP projector were trained on 5 million examples to improve image-text alignment, OCR capabilities, and text-only understanding. Data came from open-access sources.

For H2OVL Mississippi-0.8B, only the MLP projector was trained first, followed by joint optimization with the LLM, while the vision encoder remained frozen. Pretraining tasks included QA, captioning, OCR, and reasoning.

A pie chart depicting Mississippi's dataset distribution by task composition a-pie-chart-depicting-mississippi%27s-dataset-distribution-by-task-composition

Examples of image text pairs during the fine-tuning stage

H2OVL Mississippi-2B was finetuned using 12 million examples, focusing on QA, OCR, reasoning, and captioning to enhance multimodal performance. Data included both open-access and in-house sources.

H2OVL Mississippi-0.8B finetuning prioritized OCR tasks using 8 million samples, optimizing all model components.

OCR and Document QA

This image demonstrates H2OVL Mississippi's ability for OCR and document data. The image itself depicts a guide that is meant to help users comprehend an affordability calculator. this-image-demonstrates-h2ovl-mississippi%27s-ability-for-ocr-and-document-data.-the-image-itself-depicts-a-guide-that-is-meant-to-help-users-comprehend-an-affordability-calculator

Reasoning

An example of H2OVL Mississippi's capacity to reason while fine tuning for OCR and computer vision. The image used to illustrate this is as follows: Q - Examine the image's quality and provide an evaluation based on your observations. A - A clear road image shows multiple triangular traffic signs with the symbol of two pedestrians, indicating a school zone ahead. The signs are lined up along the left side of a road, gradually receding into the distance. The trees on both sides of the road are in focus, with the sky visible between them. The lighting is balanced, with vivid colors and clear textures, and the signs are well positioned, making the image quality good. an-example-of-h2ovl-mississippi%27s-capacity-to-reason-while-fine-tuning-for-ocr-and-computer-vision.-the-image-used-to-illustrate-this-is-as-follows%3A-q---examine-the-image%27s-quality-and-provide-an-evaluation-based-on-your-observations.-a---a-clear-road-image-shows-multiple-triangular-traffic-signs-with-the-symbol-of-two-pedestrians%2C-indicating-a-school-zone-ahead.-the-signs-are-lined-up-along-the-left-side-of-a-road%2C-gradually-receding-into-the-distance.-the-trees-on-both-sides-of-the-road-are-in-focus%2C-with-the-sky-visible-between-them.-the-lighting-is-balanced%2C-with-vivid-colors-and-clear-textures%2C-and-the-signs-are-well-positioned%2C-making-the-image-quality-good

Chart, figure, table understanding

This image represents H2OVL Mississippi's aptitude for understanding various chats, figures, and tables. The image itself shows a bar chart representing the accuracy of algorithms on different datasets. this-image-represents-h2ovl-mississippi%27s-aptitude-for-understanding-various-chats%2C-figures%2C-and-tables.-the-image-itself-shows-a-bar-chart-representing-the-accuracy-of-algorithms-on-different-datasets

General QA

This image depicts H2OVL Mississippi's ability for General Q and A while reference images. The image itself is a diagram illustrating a recall/broker dependency network. broker-dependency-network

Textbook, academic questions

This image represents a prompt and generated answer that's meant to illustrate H2OVL Mississippi's capacity to understand textbook questions. The image itself shows a diagram depicting a simple aquatic food chain. It starts with diatoms (green, leaf-shaped organisms), followed by midge larvae (brown segmented organism), then a brown trout (a fish), and ends with a great cormorant (a black bird with spread wings). The sequence illustrates the flow of energy from producers (diatoms) to primary consumers (midge larvae), secondary consumers (brown trout), and finally tertiary consumers (great cormorant), visually representing the concept of a food chain in an ecosystem. this-image-represents-a-prompt-and-generated-answer-that%27s-meant-to-illustrate-h2ovl-mississippi%27s-capacity-to-understand-textbook-questions.-the-image-itself-shows-a-diagram-depicting-a-simple-aquatic-food-chain.-it-starts-with-diatoms-%28green%2C-leaf-shaped-organisms%29%2C-followed-by-midge-larvae-%28brown-segmented-organism%29%2C-then-a-brown-trout-%28a-fish%29%2C-and-ends-with-a-great-cormorant-%28a-black-bird-with-spread-wings%29.-the-sequence-illustrates-the-flow-of-energy-from-producers-%28diatoms%29-to-primary-consumers-%28midge-larvae%29%2C-secondary-consumers-%28brown-trout%29%2C-and-finally-tertiary-consumers-%28great-cormorant%29%2C-visually-representing-the-concept-of-a-food-chain-in-an-ecosystem

Captioning

This image illustrates H2OVL Mississippi series ability to describe and caption images. The image itself shows a young adult woman seated in the driver's seat of a car. The light source, possibly from streetlights or another vehicle, softly illuminates her face. The woman has medium-length curly hair, and her expression appears contemplative or slightly concerned. Her hands are visible on the steering wheel, suggesting she might be driving or sitting stationary inside the vehicle. The interior of the car is dark, emphasizing her thoughtful expression. this-image-illustrates-h2ovl-mississippi-series-ability-to-describe-and-caption-images.-the-image-itself-shows-a-young-adult-woman-seated-in-the-driver%27s-seat-of-a-car.-the-light-source%2C-possibly-from-streetlights-or-another-vehicle%2C-softly-illuminates-her-face.-the-woman-has-medium-length-curly-hair%2C-and-her-expression-appears-contemplative-or-slightly-concerned.-her-hands-are-visible-on-the-steering-wheel%2C-suggesting-she-might-be-driving-or-sitting-stationary-inside-the-vehicle.-the-interior-of-the-car-is-dark%2C-emphasizing-her-thoughtful-expression

H2OVL Mississippi 2B is available worldwide

H2OVL Mississippi-0.8B is available worldwide

H2OVL Mississippi technical report