GitLab

build(docker): update doclayout-yolo dependency

- Remove doclayout_yolo==0.0.2b1 and doclayout-yolo==0.0.2
- Add doclayout-yolo==0.0.2b1 to all requirements files

update logo

fix(language): remove invalid UTF-16 surrogate pairs from input text

- Add `remove_invalid_surrogates` function to filter out invalid UTF-16 surrogate pairs
- Integrate the new function into the `detect_lang` workflow
- Include a test case with UTF-16 surrogates to verify the fix

docs(magic_pdf): update llm_aided.py prompt for title list optimization

- Clarify the expected format for the optimized title list JSON output- Emphasize the need to return only the title levels in the specified format

refactor(pre_proc): adjust IOU threshold for character overlap detection

- Modified the IOU threshold in ocr_span_list_modify.py from 0.9 to 0.35
- This change aims to improve the detection of overlapping characters in OCR processed PDFs

feat(post_proc): enhance title block processing with average line height

- Add average line height calculation for title blocks
- Include page number in title dictionary
- Improve title optimization prompt for better hierarchy- Implement retry mechanism for JSON decoding errors
- Add error logging for title count mismatch

feat(layout): improve title block handling and layout detection

- Merge title blocks that are close to each other horizontally
- Adjust line insertion logic for title blocks- Increase image size and decrease confidence threshold for layout detection
- Update DocLayoutYOLO model weights
- Refactor drawing of bounding boxes for different block types

build(deps): add upper version limit for PyMuPDF

- Set PyMuPDF version to <= 1.24.14 in all requirements files
- Prevent potential compatibility issues with future versions

Update pdf_parse_union_core_v2.py

docs/replace log

docs(faq): add troubleshooting guide for old GPUs encountering CUDA errors

Added a new section in both English and Chinese FAQs addressing the issue where old GPUs like M40 encounter a RuntimeError due to unsupported BF16 precision. The guide includes steps to manually disable BF16 precision by modifying the relevant code in "pdf_parse_union_core_v2.py".

fix: update resource URLs to jsdelivr

- Replace Gitee URLs with jsdelivr URLs in multiple files
- Update Dockerfile instructions to use new URLs
- Modify documentation to reflect new resource location
- Adjust scripts and READMEs to point to new URL for downloads

docs(README): update version number in release notes

- Change version number from 1.0.0 to1.0.1 in both English and Chinese README files
- Update the changelog section in README.md and README_zh-CN.md

master->dev

fix(llm_aided): add enable flag check for LLM aided optimizations 

fix(llm_aided): add enable flag check for LLM aided optimizations 

fix(llm_aided): add enable flag check for LLM aided optimizations

- Add enable flag check for formula, text, and title optimizations

build(docker): use Aliyun mirror for magic-pdf installation