GitLab

- Comment out OCR model initialization and execution for low-contrast spans
- Add batch OCR processing for collected image spans
- Adjust contrast threshold for OCR processing
- Remove unnecessary OCR processing for high-contrast spans
- Implement more efficient OCR workflow by processing multiple spans at once

- Remove unused OCR dictionaries for Arabic, Belarusian, Bulgarian and Armenian languages
- Update model configurations in arch_config.yaml:
- Comment out 'out_channels' for various language models
  - Rename Arabic, Korean, Japanese, Tamil and Devanagari model configurations to use 'v3' instead of 'v4'
- Delete ar_dict.txt, be_dict.txt and bg_dict.txt files
- Update arabic_dict.txt to remove blank line at the start

- Remove unused imports and code
- Simplify model architecture by removing unnecessary components
- Update initialization and forward pass logic
- Rename variables for consistency

- Added warnings module to import list
- Implemented a warning catcher to ignore FutureWarning from the transformers module
- This change prevents unnecessary warning messages during model inference

- Replace direct OCR model access with AtomModelSingleton for better model management
- Round OCR scores to 2 decimal places for consistency
- Improve error handling and logging in batch analysis
- Simplify OCR result processing in pdf_parse_union_core_v2.py

- Add support for multiple languages in OCR processing
- Create separate lists for each language to improve processing efficiency
- Update OCR model initialization to use PytorchPaddleOCR instead of ModifiedPaddleOCR
- Modify get_ocr_result_list function to include language information- Improve logging for OCR detection and recognition

- Split OCR process into detection and recognition stages
- Update batch analysis and document analysis pipelines
- Modify OCR result formatting and handling
- Remove unused imports and optimize code structure

- Add base model structure for OCR in pytorch
- Implement data augmentation and transformation modules
- Create utilities for dictionary handling and state dict conversion
- Include post-processing modules for OCR
- Add weight initialization and loading functions

feat: remove old inference code

feat: batch inference with ocr and lang flag

refactor(pdf_parse): adjust line calculation for block height

- Remove unnecessary addition of 1 when calculating lines for block height
- This change affects the logic for both potential double-column and triple-column structures

refactor(pdf_parse): adjust line calculation for block height

- Remove unnecessary addition of 1 when calculating lines for block height
- This change affects the logic for both potential double-column and triple-column structures

fix(pre_proc): improve character overlap handling in OCR processing

- Add condition to check for identical or space characters when resolving overlaps
- Skip non-conflicting character pairs to prevent unnecessary removals

fix: support auto method and auto lang

fix(magic_pdf): improve image resizing and padding in UnimerSwinn model

- Comment out margin cropping to prevent errors with broken files
- Refactor image resizing to preserve aspect ratio
- Update padding calculation and application using OpenCV

refactor(ocr): improve ONNX model initialization and resource handling

- Replace deprecated importlib.resources.path with importlib.resources.files
- Simplify code structure and improve readability
- Remove unnecessary comments and empty lines

feat(pre_proc): add function to remove x-overlapping characters in spans

- Implement `remove_x_overlapping_chars` function in `ocr_span_list_modify.py`
- Integrate the new function in `pdf_parse_union_core_v2.py` to process spans
- Remove unnecessary character replacement functions and comments

- Comment out LayoutLMv3, TableMaster, and StructEqTable models
- Update MFR model path to unimernet_hf_small_2503- Remove unused import in Unimernet.py

Dev push

- Remove separate condition for GPU memory >= 24GB
- Simplify logic to use a single threshold of 16GB

- Increase batch ratio to 32 for GPU memory >= 24GB
- Set batch ratio to 16 for GPU memory >= 16GB
- Reduce batch ratio to 8 for GPU memory >= 12GB
- Lower batch ratio to 4 for GPU memory >= 8GB
- Set batch ratio to 2 for GPU memory >= 6GB
- Keep batch ratio at 1 for lower GPU memory sizes

- Add bf_16_support check for CUDA and MPS devices
- Use bfloat16 precision for layoutreader model on supported devices
- Improve performance on devices with bf_16 support

refactor: remove torchtext deprecation warning handling

- Remove torchtext version check and deprecation warning handling from multiple files
- This code was unnecessary and potentially caused issues when torchtext was not installed

refactor(magic_pdf): remove unnecessary half() calls for CPU devices

- Remove half() calls for DocLayoutYOLO and YOLOv8 models
- This change prevents potential errors when running models on CPU

build(docker&setup): add ftfy package