GitLab

Release 0.10.3

refactor(ocr): Fix the error of paddleocr failing to initialize in a multi-threaded environment

refactor(pdf_parse): adjust character-axis alignment algorithm

- Introduce `span_height_radio` parameter to calculate_char_in_span function
- Replace fixed ratio with dynamic ratio for character and span axis alignment
- Improve flexibility and accuracy of character placement within spans

fix(ocr_mkcontent): handle empty paragraphs on pages

- Add empty paragraph handling for pages with no content
- Append an empty markdown object when a page has no paragraphs
- Increment page number even if no content is present

feat(pdf_parse): add line start flag detection and optimize line stop flag logic

- Add LINE_START_FLAG tuple to identify starting flags of a line
- Modify calculate_char_in_span function to handle both line start and stop flags
- Remove redundant char_is_line_stop_flag variable and simplify logic
- Improve line flag detection to enhance text extraction accuracy

refactor(pdf_check): improve character detection using PyMuPDF

- Replace pdfminer with PyMuPDF for character detection
- Implement new method detect_invalid_chars_by_pymupdf
- Update check_invalid_chars in pdf_meta_scan.py to use new method
- Add __replace_0xfffd function in pdf_parse_union_core_v2.py to handle special characters
- Remove unused imports and update requirements.txt

refactor(ocr): improve text processing and span handling

- Remove unused language detection code
- Simplify text content processing logic
- Update span sorting and text extraction in pdf_parse_union_core_v2.py

feat(pdf_parse): filter out skewed text lines

- Add direction filtering to ignore highly skewed text lines
- Improve text extraction accuracy by focusing on non-skewed content

refactor(para): improve language detection and block splitting

- Add language detection for each block of text
- Implement language-specific logic for right margin alignment
- Introduce logging for debugging purposes

fix(Hybrid OCR):Enable Hybrid OCR for Empty Spans That Contain a Certain Number of Placeholders but No Actual Text

fix(Hybrid OCR):Enable Hybrid OCR for Empty Spans That Contain a Certain Number of Placeholders but No Actual Text

fix(lite_model): Adapt iite Mode to the Hybrid OCR Mode in Version 0.10

master -> dev

Release 0.10.2

refactor(pdf_parse_union_core_v2): optimize page processing time logging

Feat/add s3 read write example

docs(README): remove code examples and redirect to documentation

- Remove command line and API code examples from README files
- Add links to online documentation for command line and API usage
- Update content to point users to the new locations for detailed information

refactor(ocr): remove unused functions and optimize OCR processing loop

- Remove unused function `calculate_angle_degrees`- Refactor `calculate_is_angle` to use directly in OCR processing
- Eliminate unnecessary loop index `idx` in OCR processing loops

- Remove commented-out code in ocr_dict_merge.py
- Improve imports and code organization in ocr_detect_all_bboxes.py
- Delete unnecessary empty lines and improve code readability

refactor(libs): remove unused imports and functions

- Remove unused imports from commons.py
- Delete unused functions related to AWS and S3 operations
- Update import statements in other modules to reflect changes in commons.py
- Remove redundant code and improve code readability

test: json minify

fix: test_tools unittest