GitLab

- Optimize paragraph splitting algorithm for better text block separation
- Update fast-langdetect dependency to ensure compatibility

- Increase batch ratio to 8 for GPU memory >=16GB
- Improve inference performance on systems with higher GPU memory

- Update OCR dictionary merge logic to include text blocks when processing interline equations
- This change improves the handling of equations that may be embedded within text content

- Simplify batch ratio logic for GPU memory >= 16GB
- Remove unnecessary conditions for 20GB and 40GB memory

- Simplify batch ratio logic for GPU memory >= 16GB
- Remove unnecessary conditions for 20GB and 40GB memory

- Sort detected images by area before processing to enhance MFR accuracy
- Implement stable sorting to maintain original order of images with equal

- Comment out @measure_time decorator for txt_spans_extract_v2 and sort_lines_by_model functions
- Remove logger.info for page_process_time
- Comment out PerformanceStats.print_stats call

- Add performance_stats module to measure and print execution time statistics
- Implement measure_time decorator to track execution time of key functions
- Remove multi-threading in pdf parsing for better resource management
- Optimize pdf parsing logic for improved performance

- Add ThreadPoolExecutor to process PDF pages in parallel
- Create separate function for page processing to improve readability and maintainability
- Include error handling for individual page processing tasks
- Log total page processing time for performance monitoring

- Update condition to only convert full-width letters and numbers
- Remove separate case for full-width space

- Replace complex device selection logic with a single line using torch.device
- Remove redundant checks and imports for better readability and maintainability

- Remove supports_bfloat16 variable and related checks
- Remove model.bfloat16() call for LayoutLMv3ForTokenClassification
- Simplify device selection logic

- Implement full_to_half function to convert full-width characters to half-width
- Apply conversion to span content before merging paragraphs
- Improve text processing for better readability and consistency

- Move batch model initialization outside the loop
- Collect page dimensions before analyzing- Update page info dictionary structure
- Add null dimensions for non-analyzed pages

- Introduce span_block_type_compatible function to check compatibility between span and block types
- Update fill_spans_in_blocks function to use the new compatibility check
- Improve accuracy of span allocation to blocks based on content type

- Update the logic for determining `end_page_id` to handle negative values
- This change ensures proper behavior when `end_page_id` is set to -1 or other negative values

- Add license ID information to the log for better traceability
- Improve logging format to include both license ID and expiration date

- Add ImportError handling to silence known import-related exceptions
- Improve generic exception handling to log error messages- Maintain existing specific exception handlers for license-related issues

- Add license verification logic for Ascend plugin
- Handle different license-related exceptions with appropriate error messages
- Log success message with license expiration date if verification passes
- Fall back to CPU model if license verification fails or plugin is not available

- Update instructions for AI-generated titles optimization
- Use ast.literal_eval() instead of json.loads() for parsing completion content
- Refactor variable names and logging for better code readability- Add error handling for JSON decoding issues

- Move environment variable settings for NPU, MPS, and other configurations to the global scope in doc_analyze_by_custom_model.py
- Remove redundant environment variable settings in pdf_extract_kit.py
- This change ensures consistent configuration across the application and avoids potential conflicts or duplicate settings

- Remove redundant imports for StructTableModel and TableMasterPaddleModel
- Reorder imports to group related modules together
- Update import structure for better readability and maintainability

- Remove unused utility functions
- Update import statements for better readability
- Add conditional imports for Ascend plugin
- Refactor table model initialization to support NPU

- Update calculate_contrast function to support both RGB and BGR image modes
- Add input validation for image mode in calculate_contrast function
- Modify usage of calculate_contrast function in OCR processing to specify image mode

- Increase batch size from 8 to 256 for language detection inference
- Add timing measurement for language detection process

- Rename empty_spans to need_ocr_spans for better clarity
- Add calculate_contrast function to measure image contrast
- Filter out low-contrast spans to improve OCR accuracy
- Update OCR processing workflow to use new filtering method