- 11 Dec, 2024 1 commit
-
-
icecraft authored
-
- 03 Dec, 2024 2 commits
- 21 Nov, 2024 1 commit
-
-
myhloli authored
- Check if language string is empty and set it to None - This prevents potential errors when an empty language string is passed
-
- 19 Nov, 2024 1 commit
-
-
icecraft authored
-
- 18 Nov, 2024 1 commit
-
-
icecraft authored
-
- 14 Nov, 2024 1 commit
-
-
myhloli authored
fix(parse_pipeline): Resolve post-processing exceptions caused by partial PDFs due to file corruption or non-standard format by forcing a re-print.
-
- 24 Oct, 2024 1 commit
-
-
icecraft authored
feat: add Data api
-
- 23 Oct, 2024 1 commit
-
-
myhloli authored
- Add new layout model option: DocLayout-YOLO - Implement model initialization and prediction for DocLayout-YOLO - Update configuration options to include new model- Modify existing code to support both LayoutLMv3 and DocLayout-YOLO models - Update Gradio app to support more Custom Switch
-
- 27 Sep, 2024 4 commits
-
-
myhloli authored
Refactor the draw bbox functions by removing unused imports and simplifying the code logic for drawing layout and line sorting bounding boxes. Adjust the debug configuration to enable content list dumping and disable markdown making mode.
-
myhloli authored
Introduce an additional argument `draw_bbox` in the `draw_bbox_with_number` function to enable toggling the drawing of bounding boxes on or off. When set to `False`, no bounding box will be drawn, allowing for situations where only text
-
myhloli authored
Remove debug code related to layout bbox visualization and adjust drawing functions to support optional line sorting bboxes. This change includes the removal of `draw_layout_bbox` function and updates to `draw_bbox_with_number` to support variable line width for bbox drawing.
-
myhloli authored
Add a new function `draw_line_sort_bbox` to visualize the sorting of lines on each page. This includes indexing lines and handling both text and non-text elements such as tables and images for better content organization. Also, comment out GPU-related code for flexibility and remove overlaps in bounding box detection, which improves the accuracy of layout splitting.
-
- 25 Sep, 2024 1 commit
-
-
myhloli authored
Implement a new function `draw_layout_sort_bbox` in `draw_bbox.py` to visualize the layout sorting results using the `LayoutLMv3ForTokenClassification` model. This function predicts the order of layout elements and draws them in the sorted sequence on the PDF pages.
-
- 12 Sep, 2024 3 commits
-
-
icecraft authored
-
icecraft authored
fix: 1. resolve uncorrect pair relation of figure and footnote, 2. resolve uncorrect pair relation of table and caption #590
-
icecraft authored
fix: 1. resolve uncorrect pair relation of figure and footnote, 2. resolve uncorrect pair relation of table and caption #590
-
- 10 Sep, 2024 1 commit
-
-
drunkpig authored
* Update README_zh-CN.md (#404) (#409) correct FAQ url Co-authored-by:
sfk <18810651050@163.com> * add dockerfile (#189) Co-authored-by:
drunkpig <60862764+drunkpig@users.noreply.github.com> * Update cla.yml * Update cla.yml * fix(ocr_mkcontent): improve language detection and content formatting (#458) Optimize the language detection logic to enhance content formatting. This change addresses issues with long word segmentation. Language detection now uses a threshold to determine the language of a text based on the proportion of English characters. Formatting rules for content have been updated to consider a list of languages (initially including Chinese, Japanese, and Korean) where no space is added between content segments for inline equations and text spans, improving the handling of Asian languages. The impact of these changes includes improved accuracy in language detection, better segmentation of long words, and more appropriate spacing in content formatting for multiple languages. * fix(self_modify): merge detection boxes for optimized text region detection (#448) Merge adjacent and overlapping detection boxes to optimize text region detection in the document. Post processing of text boxes is enhanced by consolidating them into larger text lines, taking into account their vertical and horizontal alignment. This improvement reduces fragmentation and improves the readability of detected text blocks. * fix(pdf-extract): adjust box threshold for OCR detection (#447) Tuned the detection box threshold parameter in the OCR model initialization to improve the accuracy of text extraction from images. The threshold was modified from 0.6 to 0.3 to filter out smaller detection boxes, which is expected to enhance the quality of the extracted text by reducing noise and false positives in the OCR process. * feat: rename the file generated by command line tools (#401) * feat: rename the file generated by command line tools * feat: add pdf filename as prefix to {span,layout,model}.pdf --------- Co-authored-by:
icecraft <tmortred@gmail.com> Co-authored-by:
icecraft <xurui1@pjlab.org.cn> * fix(ocr_mkcontent): revise table caption output (#397) * fix(ocr_mkcontent): revise table caption output - Ensuring that table captions are properly included in the output. - Remove the redundant `table_caption` variable。 * Update cla.yml * Update bug_report.yml * feat(cli): add debug option for detailed error handling Enable users to invoke the CLI command with a new debug flag to get detailed debugging information. * fix(pdf-extract-kit): adjust crop_paste parameters for better accuracyThe crop_paste_x and crop_paste_y values in the pdf_extract_kit.py have been modified to improve the accuracy and consistency of OCR processing. The new values are set to 25 to ensure more precise image cropping and pasting which leads to better OCR recognition results. * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * build(docker): update docker build step (#471) * build(docker): update base image to Ubuntu 22.04 and install PaddlePaddleUpgrade the Docker base image from ubuntu:latest to ubuntu:22.04 for improved performance and stability. Additionally, integrate PaddlePaddle GPU version 3.0.0b1 into the Docker build for enhanced AI capabilities. The MinIO configuration file has also been updated to the latest version. * build(dockerfile): Updated the Dockerfile * build(Dockerfile): update Dockerfile * docs(docker): add instructions for quick deployment with Docker Include Docker-based deployment instructions in the README for both English and Chinese locales. This update provides users a quick-start guide to using Docker for deployment, with notes on GPU VRAM requirements and default acceleration features. * build(docker): Layer the installation of dependencies, downloading the model, and the setup of the program itself. * build(docker): Layer the installation of dependencies, downloading the model, and the setup of the program itself. * upload an introduction about chemical formula and update readme.md (#489) * upload an introduction about chemical formula * rename 2 files * update readme.md at TODO in chemstery * rename 2 files and update readme.md at TODO in chemstery * update README_zh-CN.md at TODO in chemstery * upload an introduction about chemical formula and update readme.md (#489) * upload an introduction about chemical formula * rename 2 files * update readme.md at TODO in chemstery * rename 2 files and update readme.md at TODO in chemstery * update README_zh-CN.md at TODO in chemstery * fix: remove the default value of output option in tools/cli.py and tools/cli_dev.py (#494) Co-authored-by:
quyuan <quyuan@pjlab.org> * Update cla.yml * Update gpu-ci.yml * Update cli.yml * Delete .github/workflows/gpu-ci.yml * fix(pdf-parse-union-core): #492 decrease span threshold for block filling (#500) Reduce the span threshold used in fill_spans_in_blocks from 0.6 to 0.3 to improve the accuracy of block filling based on layout analysis. * fix(detect_all_bboxes): remove small overlapping blocks by merging (#501) Previously, small blocks that overlapped with larger ones were merely removed. This fix changes the approach to merge smaller blocks into the larger block instead, ensuring that no information is lost and the larger block encompasses all the text content fully. * feat(cli&analyze&pipeline): add start_page and end_page args for pagination (#507) * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * Feat/support rag (#510) * Create requirements-docker.txt * feat: update deps to support rag * feat: add support to rag, add rag_data_reader api for rag integration * feat: let user retrieve the filename of the processed file * feat: add projects demo for rag integrations --------- Co-authored-by:
Xiaomeng Zhao <moe@myhloli.com> Co-authored-by:
Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by:
liukaiwen <liukaiwen@pjlab.org.cn> * @Matthijz98 has signed the CLA in opendatalab/MinerU#467 * Create download_models.py * Create requirements-docker.txt * feat<table model>: add tablemaster with paddleocr to detect and recognize table * @strongerfly has signed the CLA in opendatalab/MinerU#487 * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table --------- Co-authored-by:
yyy <102640628+dt-yy@users.noreply.github.com> Co-authored-by:
wangbinDL <wangbin_research@163.com> * feat<table model>: add tablemaster with paddleocr to detect and recognize table (#511) * Update cla.yml * Update bug_report.yml * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
Kaiwen Liu <lkw_buaa@163.com> Co-authored-by:
icecraft <tmortred@163.com> Co-authored-by:
Siyu Hao <131659128+GDDGCZ518@users.noreply.github.com> Co-authored-by:
-
- 09 Sep, 2024 1 commit
-
-
myhloli authored
Pass the `lang` parameter to `custom_model_init` in `doc_analyze` to support language-specific OCR configurations. This enhancement allows the use of language information to improve OCR accuracy when processing PDFs.
-
- 30 Aug, 2024 1 commit
-
-
Xiaomeng Zhao authored
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system.
-
- 20 Aug, 2024 2 commits
-
-
Xiaomeng Zhao authored
* fix(ocr_mkcontent): revise table caption output - Ensuring that table captions are properly included in the output. - Remove the redundant `table_caption` variable。 * Update cla.yml * Update bug_report.yml * feat(cli): add debug option for detailed error handling Enable users to invoke the CLI command with a new debug flag to get detailed debugging information. * fix(pdf-extract-kit): adjust crop_paste parameters for better accuracyThe crop_paste_x and crop_paste_y values in the pdf_extract_kit.py have been modified to improve the accuracy and consistency of OCR processing. The new values are set to 25 to ensure more precise image cropping and pasting which leads to better OCR recognition results. * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
-
icecraft authored
* feat: rename the file generated by command line tools * feat: add pdf filename as prefix to {span,layout,model}.pdf --------- Co-authored-by:
-
- 09 Aug, 2024 1 commit
-
-
myhloli authored
Implement the feature to draw bounding boxes for model elements in the PDF. This includes adding new drawing functions and modifying existing ones to accommodate the new feature. Also, updates are made to CLI tools and common utilities to support the model bbox drawing.
-
- 04 Aug, 2024 1 commit
-
-
myhloli authored
-
- 01 Aug, 2024 1 commit
-
-
icecraft authored
* feat: refractor cli command * feat: add docs to describe the output files of cli * feat: resove review comments * feat: updat docs about middle.json --------- Co-authored-by:shenguanlin <shenguanlin@pjlab.org.cn>
-
