GitLab

refactor(txt_parse): improve text extraction accuracy with new algorithm

- Implement new text extraction method (txt_spans_extract_v2) to enhance accuracy
- Add character filling in spans for better text reconstruction
- Introduce empty span handling using OCR for missed text
- Optimize span filtering and overlap removal

feat(ocr): improve text detection and OCR accuracy 

# Conflicts:
#	magic_pdf/model/pdf_extract_kit.py

- Update OCR utils to handle different box formats and improve angle calculation
- Modify PDF extraction kit to support OCR option and optimize processing flow
- Enhance PPOCR model to sort and filter detection boxes, improving text splitting accuracy

fix(remove_overlaps_min_spans): optimize overlap detection in OCR span list modification

- Improve logic to skip dropped spans in overlap detection
- Enhance efficiency by avoiding unnecessary comparisons

fix(ocr_mkcontent): improve hyphen handling at line ends

- fix the bug where hyphens in the middle of a line are being discarded

refactor(ocr_dict_merge): add threshold parameter for line merging

- Add threshold parameter to merge_spans_to_line function
- Make threshold configurable for y-axis overlap check
- Improve flexibility and accuracy of line merging algorithm

fix(tools): handle empty language string in common.py

- Check if language string is empty and set it to None
- This prevents potential errors when an empty language string is passed

refactor: move some constants or enums defs to config folder

fix: using new data api replace old rw api

refactor(para): adjust right margin threshold based on block width

- Introduce a variable threshold for right margin based on block width
- Use 0.26 * block_weight for wider blocks (block_weight_radio >= 0.5)
- Use 0.36 * block_weight for narrower blocks- This change aims to improve paragraph splitting accuracy for different block widths

build(setup): add old_linux specific dependencies

- Add albumentations package with version <=1.4.20 for old_linux
- This version is compatible with Linux systems from 2019 and earlier
- Version 1.4.21 and above introduced simsimd which is not supported on older Linux systems

refactor(para): improve paragraph splitting logic

- Add page size information to blocks
- Calculate block width ratio relative to page width
- Adjust threshold for determining right side indentation
- Implement additional checks for merging blocks across pages
- Improve logic for identifying list structures

feat(ocr): improve handling of angled text boxes

- Add calculate_is_angle function to detect angled text boxes
- Update update_det_boxes and merge_det_boxes functions to handle angled text boxes
- Modify angle detection logic in various parts of the code

refactor(tests): extract common test utilities into test_commons.py

test(unitest): Restore unit test cases

update ci