GitLab

master->dev

Release 0.10.5

Dev -> 0.10.5

fix(pre_proc): prevent errors when imageWriter is None

refactor(para): adjust line height multiplier for block splitting,fix(pre_proc): prevent errors when imageWriter is None

- Decrease the maximum width and height from 9000 to 4500 pixels
- This change aims to prevent excessive resource usage when rendering PDFs

- Updated cut_image.py to check for NoneType imageWriter
- Prevents AttributeError when imageWriter is not provided

refactor(para): adjust line height multiplier for block splitting

- Decrease the line height multiplier from 0.8 to 0.7 for both left and right sides
- This modification aims to improve the accuracy of paragraph splitting

Release 0.10.4

fix(mkcontent): optimize paragraph text merging and language detection

- Remove overlap between bboxes for block separation
- Sort bboxes by combined x and y coordinates for better layout handling
- Comment out previous overlap removal function

- Extract language detection to block level instead of line level
- Improve logic for handling Chinese, Japanese, and Korean languages
- Refactor code for better readability and performance
- Optimize handling of hyphenated words at line ends

- Introduce language detection to determine line spacing based on language context
- Implement different spacing rules for Chinese/Japanese/Korean and Western texts
- Adjust span content handling based on detected language and span type

master->dev

Release 0.10.3

refactor(ocr): Fix the error of paddleocr failing to initialize in a multi-threaded environment

refactor(pdf_parse): adjust character-axis alignment algorithm

- Introduce `span_height_radio` parameter to calculate_char_in_span function
- Replace fixed ratio with dynamic ratio for character and span axis alignment
- Improve flexibility and accuracy of character placement within spans

fix(ocr_mkcontent): handle empty paragraphs on pages

- Add empty paragraph handling for pages with no content
- Append an empty markdown object when a page has no paragraphs
- Increment page number even if no content is present

feat(pdf_parse): add line start flag detection and optimize line stop flag logic

- Add LINE_START_FLAG tuple to identify starting flags of a line
- Modify calculate_char_in_span function to handle both line start and stop flags
- Remove redundant char_is_line_stop_flag variable and simplify logic
- Improve line flag detection to enhance text extraction accuracy

refactor(pdf_check): improve character detection using PyMuPDF

- Replace pdfminer with PyMuPDF for character detection
- Implement new method detect_invalid_chars_by_pymupdf
- Update check_invalid_chars in pdf_meta_scan.py to use new method
- Add __replace_0xfffd function in pdf_parse_union_core_v2.py to handle special characters
- Remove unused imports and update requirements.txt

refactor(ocr): improve text processing and span handling