ThirdBrAIn.tech

ByteDance Dolphin - Document Image Parsing Model - Install and Test Locally

2 min read

ByteDance Dolphin - Document Image Parsing Model - Install and Test Locally

AI Summary

This video demonstrates the installation and testing of ByteDance’s Dolphin model, a document image parsing AI that uses a two-step “analyze-then-parse” approach to extract content from complex documents containing text, images, tables, and formulas.

Key Features of Dolphin Model

Architecture:

  • Vision Encoder: Uses Swin Transformer to examine document images and extract visual information
  • Text Decoder: Uses mBART for multilingual text conversion and structured output
  • Two-step process: First analyzes document layout and structure, then extracts specific content

Capabilities:

  • Extracts text, tables, formulas, and other elements from document images
  • Works with plain English or Chinese instructions
  • Outputs results in both Markdown and JSON formats
  • Runs efficiently on CPU (only uses ~2GB VRAM briefly during processing)

Installation Process

The video shows a complete local installation on Ubuntu with NVIDIA RTX 6000:

  1. Create virtual environment
  2. Clone the Dolphin repository from HuggingFace
  3. Install requirements using HuggingFace CLI
  4. Login to HuggingFace and download the model
  5. Run test scripts on sample documents

Performance Testing

The presenter tests Dolphin on various document elements:

Full Page Text Extraction: Successfully extracted complete document text with high accuracy, comparable to DocLing quality

Table Extraction: Accurately converted complex tables to Markdown format with proper formatting, brackets, and dots preserved

Formula Extraction: Successfully extracted mathematical formulas in proper LaTeX/boxed format

Paragraph Extraction: Good performance with minor navigation issues between paragraphs, but overall accurate text extraction

Key Advantages

  • Speed: Lightning-fast processing (7 seconds or less per operation)
  • Resource Efficiency: Can run on CPU, minimal VRAM usage
  • Output Formats: Dual output in Markdown and structured JSON
  • Element-Specific Extraction: Can target specific document elements (tables, formulas, text)
  • Quality: Performance comparable to IBM’s DocLing model

The video concludes that Dolphin represents excellent work from ByteDance, positioning it as a strong competitor to existing document parsing solutions like DocLing, with particular strengths in speed and resource efficiency.

Graph View