- 26 Nov, 2024 1 commit
-
-
myhloli authored
-
- 21 Nov, 2024 2 commits
-
-
myhloli authored
- Improve logic to skip dropped spans in overlap detection - Enhance efficiency by avoiding unnecessary comparisons
-
myhloli authored
- Add threshold parameter to merge_spans_to_line function - Make threshold configurable for y-axis overlap check - Improve flexibility and accuracy of line merging algorithm
-
- 19 Nov, 2024 1 commit
-
-
icecraft authored
-
- 25 Oct, 2024 2 commits
-
-
myhloli authored
- Lower the Y-axis overlap threshold for merging spans into lines from0.6 to 0.5 - Reduce the unclip ratio for OCR detection from 2.4 to 1.8
-
myhloli authored
- Split image and table blocks into separate categories - Add group_id to image and table blocks- Update block processing logic to handle new categories - Modify layout splitting and span filling to accommodate new block types - Adjust block indexing and sorting to consider new structures
-
- 23 Oct, 2024 1 commit
-
-
myhloli authored
- Add new layout model option: DocLayout-YOLO - Implement model initialization and prediction for DocLayout-YOLO - Update configuration options to include new model- Modify existing code to support both LayoutLMv3 and DocLayout-YOLO models - Update Gradio app to support more Custom Switch
-
- 14 Oct, 2024 1 commit
-
-
myhloli authored
- Add detection for list and index blocks in OCR processing- Implement merging of list and index blocks across pages - Update block types to include list and index categories - Adjust text merging logic to handle new block types - Modify layout drawing to distinguish list and index blocks
-
- 27 Sep, 2024 2 commits
-
-
myhloli authored
Add a new function `draw_line_sort_bbox` to visualize the sorting of lines on each page. This includes indexing lines and handling both text and non-text elements such as tables and images for better content organization. Also, comment out GPU-related code for flexibility and remove overlaps in bounding box detection, which improves the accuracy of layout splitting.
-
myhloli authored
refactor(pdf_parse_union): integrate LayoutLMv3 for block orderingReplace the heuristic-based block ordering algorithm with LayoutLMv3 model predictions toimprove the accuracy of block ordering on PDF pages. Additionally, refactor the span handling during block filling to ensure spans are correctly assigned. - Introduce LayoutLMv3ForTokenClassification from 'hantian/layoutreader' to predict block order. - Implement span replacement strategy to use pymu spans for non-OCR content. - Enhance cleanup process to free GPU memory more effectively after model use. - Adjust block ordering logic to use median line index for text, title, and interline equation blocks. - Refactor page parsing core logic for better maintainability. BREAKING CHANGE: The integration of LayoutLMv3 changes the internal block handling and ordering mechanism, which may affect downstream systems relying on the previous implementation. Ensure to test thoroughly before deployment.
-
- 10 Sep, 2024 1 commit
-
-
drunkpig authored
* Update README_zh-CN.md (#404) (#409) correct FAQ url Co-authored-by:
sfk <18810651050@163.com> * add dockerfile (#189) Co-authored-by:
drunkpig <60862764+drunkpig@users.noreply.github.com> * Update cla.yml * Update cla.yml * fix(ocr_mkcontent): improve language detection and content formatting (#458) Optimize the language detection logic to enhance content formatting. This change addresses issues with long word segmentation. Language detection now uses a threshold to determine the language of a text based on the proportion of English characters. Formatting rules for content have been updated to consider a list of languages (initially including Chinese, Japanese, and Korean) where no space is added between content segments for inline equations and text spans, improving the handling of Asian languages. The impact of these changes includes improved accuracy in language detection, better segmentation of long words, and more appropriate spacing in content formatting for multiple languages. * fix(self_modify): merge detection boxes for optimized text region detection (#448) Merge adjacent and overlapping detection boxes to optimize text region detection in the document. Post processing of text boxes is enhanced by consolidating them into larger text lines, taking into account their vertical and horizontal alignment. This improvement reduces fragmentation and improves the readability of detected text blocks. * fix(pdf-extract): adjust box threshold for OCR detection (#447) Tuned the detection box threshold parameter in the OCR model initialization to improve the accuracy of text extraction from images. The threshold was modified from 0.6 to 0.3 to filter out smaller detection boxes, which is expected to enhance the quality of the extracted text by reducing noise and false positives in the OCR process. * feat: rename the file generated by command line tools (#401) * feat: rename the file generated by command line tools * feat: add pdf filename as prefix to {span,layout,model}.pdf --------- Co-authored-by:
icecraft <tmortred@gmail.com> Co-authored-by:
icecraft <xurui1@pjlab.org.cn> * fix(ocr_mkcontent): revise table caption output (#397) * fix(ocr_mkcontent): revise table caption output - Ensuring that table captions are properly included in the output. - Remove the redundant `table_caption` variable。 * Update cla.yml * Update bug_report.yml * feat(cli): add debug option for detailed error handling Enable users to invoke the CLI command with a new debug flag to get detailed debugging information. * fix(pdf-extract-kit): adjust crop_paste parameters for better accuracyThe crop_paste_x and crop_paste_y values in the pdf_extract_kit.py have been modified to improve the accuracy and consistency of OCR processing. The new values are set to 25 to ensure more precise image cropping and pasting which leads to better OCR recognition results. * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * build(docker): update docker build step (#471) * build(docker): update base image to Ubuntu 22.04 and install PaddlePaddleUpgrade the Docker base image from ubuntu:latest to ubuntu:22.04 for improved performance and stability. Additionally, integrate PaddlePaddle GPU version 3.0.0b1 into the Docker build for enhanced AI capabilities. The MinIO configuration file has also been updated to the latest version. * build(dockerfile): Updated the Dockerfile * build(Dockerfile): update Dockerfile * docs(docker): add instructions for quick deployment with Docker Include Docker-based deployment instructions in the README for both English and Chinese locales. This update provides users a quick-start guide to using Docker for deployment, with notes on GPU VRAM requirements and default acceleration features. * build(docker): Layer the installation of dependencies, downloading the model, and the setup of the program itself. * build(docker): Layer the installation of dependencies, downloading the model, and the setup of the program itself. * upload an introduction about chemical formula and update readme.md (#489) * upload an introduction about chemical formula * rename 2 files * update readme.md at TODO in chemstery * rename 2 files and update readme.md at TODO in chemstery * update README_zh-CN.md at TODO in chemstery * upload an introduction about chemical formula and update readme.md (#489) * upload an introduction about chemical formula * rename 2 files * update readme.md at TODO in chemstery * rename 2 files and update readme.md at TODO in chemstery * update README_zh-CN.md at TODO in chemstery * fix: remove the default value of output option in tools/cli.py and tools/cli_dev.py (#494) Co-authored-by:
quyuan <quyuan@pjlab.org> * Update cla.yml * Update gpu-ci.yml * Update cli.yml * Delete .github/workflows/gpu-ci.yml * fix(pdf-parse-union-core): #492 decrease span threshold for block filling (#500) Reduce the span threshold used in fill_spans_in_blocks from 0.6 to 0.3 to improve the accuracy of block filling based on layout analysis. * fix(detect_all_bboxes): remove small overlapping blocks by merging (#501) Previously, small blocks that overlapped with larger ones were merely removed. This fix changes the approach to merge smaller blocks into the larger block instead, ensuring that no information is lost and the larger block encompasses all the text content fully. * feat(cli&analyze&pipeline): add start_page and end_page args for pagination (#507) * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * Feat/support rag (#510) * Create requirements-docker.txt * feat: update deps to support rag * feat: add support to rag, add rag_data_reader api for rag integration * feat: let user retrieve the filename of the processed file * feat: add projects demo for rag integrations --------- Co-authored-by:
Xiaomeng Zhao <moe@myhloli.com> Co-authored-by:
Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by:
liukaiwen <liukaiwen@pjlab.org.cn> * @Matthijz98 has signed the CLA in opendatalab/MinerU#467 * Create download_models.py * Create requirements-docker.txt * feat<table model>: add tablemaster with paddleocr to detect and recognize table * @strongerfly has signed the CLA in opendatalab/MinerU#487 * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table --------- Co-authored-by:
yyy <102640628+dt-yy@users.noreply.github.com> Co-authored-by:
wangbinDL <wangbin_research@163.com> * feat<table model>: add tablemaster with paddleocr to detect and recognize table (#511) * Update cla.yml * Update bug_report.yml * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
Kaiwen Liu <lkw_buaa@163.com> Co-authored-by:
icecraft <tmortred@163.com> Co-authored-by:
Siyu Hao <131659128+GDDGCZ518@users.noreply.github.com> Co-authored-by:
-
- 03 Sep, 2024 1 commit
-
-
icecraft authored
* feat: support figure footnote * feat: using the relative position to combine footnote, table, image * feat: add the readme of projects * fix: code spell in unittest --------- Co-authored-by:
-
- 28 Aug, 2024 1 commit
-
-
Xiaomeng Zhao authored
Previously, small blocks that overlapped with larger ones were merely removed. This fix changes the approach to merge smaller blocks into the larger block instead, ensuring that no information is lost and the larger block encompasses all the text content fully.
-
- 23 Jul, 2024 2 commits
- 22 Jul, 2024 1 commit
-
-
myhloli authored
-
- 25 Jun, 2024 1 commit
-
-
赵小蒙 authored
-
- 07 Jun, 2024 1 commit
-
-
赵小蒙 authored
-
- 06 Jun, 2024 1 commit
-
-
赵小蒙 authored
-
- 24 May, 2024 1 commit
-
-
赵小蒙 authored
-
- 22 May, 2024 1 commit
-
-
liukaiwen authored
-
- 10 May, 2024 4 commits
- 09 May, 2024 1 commit
-
-
赵小蒙 authored
-
- 08 May, 2024 2 commits
- 07 May, 2024 1 commit
-
-
赵小蒙 authored
-
- 06 May, 2024 1 commit
-
-
赵小蒙 authored
-
- 30 Apr, 2024 1 commit
-
-
许瑞 authored
-
- 29 Apr, 2024 7 commits
- 28 Apr, 2024 1 commit
-
-
许瑞 authored
-
- 25 Apr, 2024 1 commit
-
-
赵小蒙 authored
-
