GitLab

feat(pdf_parse): remove tilted lines for better text extraction

- Add remove_tilted_line function to filter out lines with angles between 2 and 88 degrees
- Integrate the new function into the text extraction process
- Improve the accuracy of text block processing by removing non-horizontal/vertical lines

LGTM

feat(llm_aided): add reasonability check and fine-tuning guidelines

- Added instructions for checking the reasonability of heading levels
- Included guidelines for making fine adjustments based on context and logic
- Emphasized the importance of aligning the final result with the document's actual structure

fix(magic_pdf): limit batch ratio for GPU memory

- Commented out the original batch ratio calculation
- Set a fixed batch ratio of 2 for GPUs with less than 8 GB memory
- Increased batch ratio to 4 for GPUs with 8 GB or more memory

refactor(model): update config version check to 1.1.1

- Update the version check in download_models.py and download_models_hf.py
- Change the threshold from '1.1.0' to '1.1.1' for model configuration updates

refactor(table): add device configuration for Unitable model

- Import get_device function from magic_pdf.libs.config_reader- Update RapidTableModel initialization to include device parameter for Unitable model

docs(README): update WeChat group link

- Update WeChat group link in both README.md and README_zh-CN.md

feat(table): upgrade RapidTable to1.0.3 and add sub-model support

- Modify the batch analyze process to handle the rapid table model's output
- Add logic_points variable to capture additional output from rapid table prediction

- Update rapid-table version from ==0.3.0 to >=1.0.3,<2.0.0 in multiple requirements files
- This change affects Ascend NPU, China, and Global Docker configurations

- Update RapidTable dependency to version 1.0.3
- Add support for sub-models in RapidTable
- Update magic-pdf configuration to include table sub-model
- Modify table model initialization to support sub-models
- Update table prediction logic to handle new output format

docs(README): update demo badges

- Update OpenDataLab badge to new design

- Update OpenDataLab badge to new design

fix(magic_pdf): correct end page index and improve error handling

- Adjust end_page_id calculation to prevent IndexError when accessing pages
- Enhance error handling in LLM post-processing by specifically catching JSONDecodeError

refactor(magic_pdf): improve title block merging logic

- Rename and update merge_title_blocks function
- Implement merge_two_bbox helper function
- Refactor merging logic to preserve original block structure- Update function calls and integrate with existing pipeline

feat(model): improve batch analysis logic and support npu

- Add support for NPU (Neural Processing Unit) when available
- Implement batch analysis for GPU and NPU devices
- Optimize memory usage and improve performance
- Update logging and error handling

build(docker): update doclayout-yolo dependency

- Remove doclayout_yolo==0.0.2b1 and doclayout-yolo==0.0.2
- Add doclayout-yolo==0.0.2b1 to all requirements files

update logo

fix(language): remove invalid UTF-16 surrogate pairs from input text

- Add `remove_invalid_surrogates` function to filter out invalid UTF-16 surrogate pairs
- Integrate the new function into the `detect_lang` workflow
- Include a test case with UTF-16 surrogates to verify the fix

docs(magic_pdf): update llm_aided.py prompt for title list optimization

- Clarify the expected format for the optimized title list JSON output- Emphasize the need to return only the title levels in the specified format

refactor(pre_proc): adjust IOU threshold for character overlap detection

- Modified the IOU threshold in ocr_span_list_modify.py from 0.9 to 0.35
- This change aims to improve the detection of overlapping characters in OCR processed PDFs

feat(post_proc): enhance title block processing with average line height

- Add average line height calculation for title blocks
- Include page number in title dictionary
- Improve title optimization prompt for better hierarchy- Implement retry mechanism for JSON decoding errors
- Add error logging for title count mismatch