- 16 Dec, 2024 1 commit
-
-
myhloli authored
- Remove YOLO_VERBOSE environment variable from multiple files - Set verbose=False in YOLOv8 prediction method to suppress logger output
-
- 12 Dec, 2024 1 commit
-
-
myhloli authored
- Add initial setup for layout detection - Implement conditional cropping for tall images - Skip cropping for wide images to improve performance - Reuse Image object across layout detection steps
-
- 11 Dec, 2024 1 commit
-
-
myhloli authored
- Implement image cropping and pasting technique to enhance layout detection - Adjust detected polygons to original image coordinates - Add comments for better code readability
-
- 06 Dec, 2024 7 commits
-
-
myhloli authored
- Remove concurrency limit logic from app.py - Update model initialization process in various modules - Remove unused VRAM check for concurrency limit - Refactor OCR model initialization in pdf_extract_kit.py - Update txt_spans_extract_v2 function to use lang parameter instead of ocr_model
-
myhloli authored
- Remove usage of AtomModelSingleton for OCR model creation - Add ocr_model_init function to initialize OCR model - Update OCR model initialization in pdf_extract_kit.py and pdf_parse_union_core_v2.py - Modify txt_spans_extract_v2 function to accept ocr_model as a parameter - Update parse_page_core function to use ocr_model instead of lang for OCR processing
-
myhloli authored
- Add threading support for OCR model initialization - Modify AtomModelSingleton to handle thread-specific instances - Update PDFExtractKit and PDFParseUnionCoreV2 to use new thread-safe OCR initialization
-
myhloli authored
- Remove threading.Lock import and usage - Delete unused model initialization comments and code- Simplify OCR model initialization in both pdf_extract_kit.py and pdf_parse_union_core_v2.py
-
myhloli authored
- Remove usage of AtomModelSingleton for OCR model initialization - Add import of ocr_model_init from model_init module - Update OCR model initialization process to use ocr_model_init function - Remove lock for OCR processing as it's no longer needed
-
myhloli authored
- Remove usage of ModelSingleton class - Initialize model directly using custom_model_init function - Add self._lock attribute to PDFExtractKit class for thread safety- Replace local lock with self._lock for OCR processing
-
赵小蒙 authored
- Remove unnecessary threading.Lock in AtomModelSingleton - Add threading.Lock to CustomPEKModel for OCR processing - Simplify model initialization logic in AtomModelSingleton
-
- 22 Nov, 2024 1 commit
-
-
myhloli authored
- Move page total time logging to doc_analyze_by_custom_model.py - Remove page total time logging from pdf_extract_kit.py - Add page_start timing variable to custom model analysis - Update logger output format for page total time
-
- 21 Nov, 2024 1 commit
-
-
myhloli authored
- Update OCR utils to handle different box formats and improve angle calculation - Modify PDF extraction kit to support OCR option and optimize processing flow - Enhance PPOCR model to sort and filter detection boxes, improving text splitting accuracy
-
- 19 Nov, 2024 1 commit
-
-
icecraft authored
-
- 15 Nov, 2024 1 commit
-
-
myhloli authored
-
- 08 Nov, 2024 2 commits
-
-
myhloli authored
- Integrate RapidOCR with RapidTable model for table recognition - Improve memory management for devices with <= 8GB VRAM - Update table recognition process to use RapidOCR for RapidTable - Add rapidocr-paddle dependency in setup.py
-
myhloli authored
- Add RapidTable model support for table recognition - Update table model configuration and initialization - Modify table recognition process to use RapidTable when specified - Add RapidTable dependency to setup.py
-
- 06 Nov, 2024 1 commit
-
-
myhloli authored
- Remove unused code for copying detection and recognition models - Simplify OCR model initialization using atom_model_manager - Delete unnecessary comments and empty lines
-
- 04 Nov, 2024 2 commits
-
-
myhloli authored
- Update StructTableModel to use the latest struct-eqtable library - Add support for HTML table extraction in PDF Extract Kit - Improve error handling and model initialization - Update dependencies in setup.py for struct-eqtable
-
ciaran authored
Modify line 397 to ensure compatibility with CPU execution, addressing the issue where specifying 'cpu' in config.json still results in a ValueError for expecting a cuda device but getting 'cpu' during demo execution.
-
- 28 Oct, 2024 3 commits
-
-
myhloli authored
- Remove import and usage of StructTableModel- Add support for TableMaster model- Update table model initialization logic to support TableMaster - Log error and exit if StructEqTable is selected, as it's under upgrade - Update README files to reflect changes in table parsing capabilities
-
liukaiwen authored
-
liukaiwen authored
-
- 25 Oct, 2024 1 commit
-
-
myhloli authored
- Lower the Y-axis overlap threshold for merging spans into lines from0.6 to 0.5 - Reduce the unclip ratio for OCR detection from 2.4 to 1.8
-
- 24 Oct, 2024 1 commit
-
-
myhloli authored
- Changed the confidence threshold from0.15 to 0.25 in the DocLayout_YOLO model prediction - This adjustment aims to improve the accuracy of layout detection by filtering out low-confidence predictions
-
- 23 Oct, 2024 1 commit
-
-
myhloli authored
- Add new layout model option: DocLayout-YOLO - Implement model initialization and prediction for DocLayout-YOLO - Update configuration options to include new model- Modify existing code to support both LayoutLMv3 and DocLayout-YOLO models - Update Gradio app to support more Custom Switch
-
- 17 Oct, 2024 2 commits
-
-
liukaiwen authored
-
myhloli authored
- Remove unused functions such as split_long_words, ocr_mk_mm_markdown_with_para, etc. - Simplify ocr_mk_markdown_with_para_core_v2 by removing unnecessary language detection and word splitting logic- Remove wordninja dependency from requirements - Update ocr_model_init to include additional parameters for OCR model configuration
-
- 14 Oct, 2024 1 commit
-
-
myhloli authored
- Add detection for list and index blocks in OCR processing- Implement merging of list and index blocks across pages - Update block types to include list and index categories - Adjust text merging logic to handle new block types - Modify layout drawing to distinguish list and index blocks
-
- 08 Oct, 2024 2 commits
-
-
liukaiwen authored
-
myhloli authored
- Introduce a conditional memory cleanup step in the PDF extraction process - Assess available GPU memory before deciding to perform memory cleanup- Log the time taken for garbage collection when it occurs - This optimization helps to balance performance and resource utilization
-
- 06 Oct, 2024 1 commit
-
-
myhloli authored
- Enhance timing output precision to two decimal places for better readability- Calculate and log document analysis speed in pages per second - Optimize logging for YOLO and table recognition processes - Remove unnecessary comments and improve code efficiency
-
- 29 Sep, 2024 1 commit
-
-
myhloli authored
The clean_memory function has been removed from pdf_parse_union_core_v2.py due to it not being used. This change streamlines the code and prevents potential confusion regarding its purpose.
-
- 20 Sep, 2024 1 commit
-
-
myhloli authored
-
- 19 Sep, 2024 2 commits
- 12 Sep, 2024 1 commit
-
-
myhloli authored
-
- 10 Sep, 2024 3 commits
-
-
drunkpig authored
* release: release 0.7.1 version (#526) * Update README_zh-CN.md (#404) (#409) correct FAQ url Co-authored-by:
sfk <18810651050@163.com> * add dockerfile (#189) Co-authored-by:
drunkpig <60862764+drunkpig@users.noreply.github.com> * Update cla.yml * Update cla.yml * feat<table model>: add tablemaster with paddleocr to detect and recognize table (#493) * Update cla.yml * Update bug_report.yml * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by:
Xiaomeng Zhao <moe@myhloli.com> * <fix>(para_split_v2): index out of range issue of span_text first char (#396) Co-authored-by:
liukaiwen <liukaiwen@pjlab.org.cn> * @Matthijz98 has signed the CLA in opendatalab/MinerU#467 * Create download_models.py * Create requirements-docker.txt * feat<table model>: add tablemaster with paddleocr to detect and recognize table * @strongerfly has signed the CLA in opendatalab/MinerU#487 * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table --------- Co-authored-by:
github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by:
yyy <102640628+dt-yy@users.noreply.github.com> Co-authored-by:
wangbinDL <wangbin_research@163.com> * feat<table model>: add tablemaster with paddleocr to detect and recognize table (#511) * Update cla.yml * Update bug_report.yml * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
Kaiwen Liu <lkw_buaa@163.com> Co-authored-by:
-
drunkpig authored
* Update README_zh-CN.md (#404) (#409) correct FAQ url Co-authored-by:
icecraft <tmortred@gmail.com> Co-authored-by:
icecraft <xurui1@pjlab.org.cn> * fix(ocr_mkcontent): revise table caption output (#397) * fix(ocr_mkcontent): revise table caption output - Ensuring that table captions are properly included in the output. - Remove the redundant `table_caption` variable。 * Update cla.yml * Update bug_report.yml * feat(cli): add debug option for detailed error handling Enable users to invoke the CLI command with a new debug flag to get detailed debugging information. * fix(pdf-extract-kit): adjust crop_paste parameters for better accuracyThe crop_paste_x and crop_paste_y values in the pdf_extract_kit.py have been modified to improve the accuracy and consistency of OCR processing. The new values are set to 25 to ensure more precise image cropping and pasting which leads to better OCR recognition results. * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
quyuan <quyuan@pjlab.org> * Update cla.yml * Update gpu-ci.yml * Update cli.yml * Delete .github/workflows/gpu-ci.yml * fix(pdf-parse-union-core): #492 decrease span threshold for block filling (#500) Reduce the span threshold used in fill_spans_in_blocks from 0.6 to 0.3 to improve the accuracy of block filling based on layout analysis. * fix(detect_all_bboxes): remove small overlapping blocks by merging (#501) Previously, small blocks that overlapped with larger ones were merely removed. This fix changes the approach to merge smaller blocks into the larger block instead, ensuring that no information is lost and the larger block encompasses all the text content fully. * feat(cli&analyze&pipeline): add start_page and end_page args for pagination (#507) * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * Feat/support rag (#510) * Create requirements-docker.txt * feat: update deps to support rag * feat: add support to rag, add rag_data_reader api for rag integration * feat: let user retrieve the filename of the processed file * feat: add projects demo for rag integrations --------- Co-authored-by:
icecraft <tmortred@163.com> Co-authored-by:
Siyu Hao <131659128+GDDGCZ518@users.noreply.github.com> Co-authored-by:
-
myhloli authored
Update the paths to model weights and configuration files for the UniMERNet architecture in both the demo.yaml and model_configs.yaml files. Adjust the mfr_model_init function toreflect the new weight and configuration paths. The changes include specifying more detailed paths to the unimernet_base directory and changing the weight file extension to .pth.
-
- 09 Sep, 2024 1 commit
-
-
myhloli authored
Pass the `lang` parameter to `custom_model_init` in `doc_analyze` to support language-specific OCR configurations. This enhancement allows the use of language information to improve OCR accuracy when processing PDFs.
-
