- 10 Sep, 2024 2 commits
-
-
drunkpig authored
* release: release 0.7.1 version (#526) * Update README_zh-CN.md (#404) (#409) correct FAQ url Co-authored-by:
sfk <18810651050@163.com> * add dockerfile (#189) Co-authored-by:
drunkpig <60862764+drunkpig@users.noreply.github.com> * Update cla.yml * Update cla.yml * feat<table model>: add tablemaster with paddleocr to detect and recognize table (#493) * Update cla.yml * Update bug_report.yml * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by:
Xiaomeng Zhao <moe@myhloli.com> * <fix>(para_split_v2): index out of range issue of span_text first char (#396) Co-authored-by:
liukaiwen <liukaiwen@pjlab.org.cn> * @Matthijz98 has signed the CLA in opendatalab/MinerU#467 * Create download_models.py * Create requirements-docker.txt * feat<table model>: add tablemaster with paddleocr to detect and recognize table * @strongerfly has signed the CLA in opendatalab/MinerU#487 * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table --------- Co-authored-by:
github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by:
yyy <102640628+dt-yy@users.noreply.github.com> Co-authored-by:
wangbinDL <wangbin_research@163.com> * feat<table model>: add tablemaster with paddleocr to detect and recognize table (#511) * Update cla.yml * Update bug_report.yml * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
Kaiwen Liu <lkw_buaa@163.com> Co-authored-by:
-
myhloli authored
Update the paths to model weights and configuration files for the UniMERNet architecture in both the demo.yaml and model_configs.yaml files. Adjust the mfr_model_init function toreflect the new weight and configuration paths. The changes include specifying more detailed paths to the unimernet_base directory and changing the weight file extension to .pth.
-
- 09 Sep, 2024 4 commits
-
-
myhloli authored
Introduce a new parameter `lang` to the `UNIPipe` class's `parse_union_pdf` function to allow specifying the language for document analysis. This enhancement enables users to customizethe language setting for better processing of multilingual documents.
-
Kaiwen Liu authored
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination * fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384 * fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384 * fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384 * Update README.md * Update README_zh-CN.md * Update README_zh-CN.md add HF、modelscope、colab url * Update README.md * Update README.md * Update README.md * Update README.md * Update README_zh-CN.md * Rename README.md to README_zh-CN.md * Create readme.md * Rename readme.md to README.md * Rename README.md to README_zh-CN.md * Update README_zh-CN.md * Create README.md * Update README.md * Update README.md * Update README.md * Update README_zh-CN.md * Update README.md * Update README_zh-CN.md * Update README_zh-CN.md * Update README.md * Update README_zh-CN.md * fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384 --------- Co-authored-by:
-
myhloli authored
-
myhloli authored
Pass the `lang` parameter to `custom_model_init` in `doc_analyze` to support language-specific OCR configurations. This enhancement allows the use of language information to improve OCR accuracy when processing PDFs.
-
- 03 Sep, 2024 3 commits
-
-
Xiaomeng Zhao authored
Refactor the pdf_extract_kit module to utilize a singleton pattern when initializing atomic models. This change ensures that atomic models are instantiated at most once, optimizing memory usage and reducing redundant initialization steps. The AtomModelSingleton class now manages the instantiation and retrieval of atomic models, improving the overall structure and efficiency of the codebase.
-
icecraft authored
* feat: support figure footnote * feat: using the relative position to combine footnote, table, image * feat: add the readme of projects * fix: code spell in unittest --------- Co-authored-by:icecraft <xurui1@pjlab.org.cn>
-
Kaiwen Liu authored
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination * fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384 * fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384 * fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
-
- 02 Sep, 2024 4 commits
-
-
yyy authored
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#493) * Update cla.yml * Update bug_report.yml * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
-
drunkpig authored
-
Xiaomeng Zhao authored
-
drunkpig authored
* fix replace \u0002, \u0003 in common text * fix(para): When an English line ends with a hyphen, do not add a space at the end.
-
- 30 Aug, 2024 2 commits
-
-
icecraft authored
* Create requirements-docker.txt * feat: update deps to support rag * feat: add support to rag, add rag_data_reader api for rag integration * feat: let user retrieve the filename of the processed file * feat: add projects demo for rag integrations --------- Co-authored-by:
-
Xiaomeng Zhao authored
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system. * feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing pipeline to support pagination functionality. This feature allows users to specify the range of pages to be processed, enhancing the efficiency and flexibility of the system.
-
- 28 Aug, 2024 3 commits
-
-
Xiaomeng Zhao authored
Previously, small blocks that overlapped with larger ones were merely removed. This fix changes the approach to merge smaller blocks into the larger block instead, ensuring that no information is lost and the larger block encompasses all the text content fully.
-
Xiaomeng Zhao authored
Reduce the span threshold used in fill_spans_in_blocks from 0.6 to 0.3 to improve the accuracy of block filling based on layout analysis.
-
icecraft authored
Co-authored-by:
-
- 20 Aug, 2024 6 commits
-
-
Xiaomeng Zhao authored
* fix(ocr_mkcontent): revise table caption output - Ensuring that table captions are properly included in the output. - Remove the redundant `table_caption` variable。 * Update cla.yml * Update bug_report.yml * feat(cli): add debug option for detailed error handling Enable users to invoke the CLI command with a new debug flag to get detailed debugging information. * fix(pdf-extract-kit): adjust crop_paste parameters for better accuracyThe crop_paste_x and crop_paste_y values in the pdf_extract_kit.py have been modified to improve the accuracy and consistency of OCR processing. The new values are set to 25 to ensure more precise image cropping and pasting which leads to better OCR recognition results. * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by:
-
Kaiwen Liu authored
Co-authored-by: -
icecraft authored
* feat: rename the file generated by command line tools * feat: add pdf filename as prefix to {span,layout,model}.pdf --------- Co-authored-by:icecraft <tmortred@gmail.com> Co-authored-by:
-
Xiaomeng Zhao authored
Tuned the detection box threshold parameter in the OCR model initialization to improve the accuracy of text extraction from images. The threshold was modified from 0.6 to 0.3 to filter out smaller detection boxes, which is expected to enhance the quality of the extracted text by reducing noise and false positives in the OCR process.
-
Xiaomeng Zhao authored
Merge adjacent and overlapping detection boxes to optimize text region detection in the document. Post processing of text boxes is enhanced by consolidating them into larger text lines, taking into account their vertical and horizontal alignment. This improvement reduces fragmentation and improves the readability of detected text blocks.
-
Xiaomeng Zhao authored
Optimize the language detection logic to enhance content formatting. This change addresses issues with long word segmentation. Language detection now uses a threshold to determine the language of a text based on the proportion of English characters. Formatting rules for content have been updated to consider a list of languages (initially including Chinese, Japanese, and Korean) where no space is added between content segments for inline equations and text spans, improving the handling of Asian languages. The impact of these changes includes improved accuracy in language detection, better segmentation of long words, and more appropriate spacing in content formatting for multiple languages.
-
- 09 Aug, 2024 6 commits
-
-
myhloli authored
-
myhloli authored
Implement the feature to draw bounding boxes for model elements in the PDF. This includes adding new drawing functions and modifying existing ones to accommodate the new feature. Also, updates are made to CLI tools and common utilities to support the model bbox drawing.
-
myhloli authored
Previously, images were not enlarged if their width or height exceeded 3000 pixels. This threshold has been increased to 9000 pixels to better handle high-resolutionscans and improve the analysis of documents with larger dimensions.
-
myhloli authored
fix(pdf-extract-kit): ensure table extraction success with additional ending conditionAdd an additional condition to determine the success of table extraction by checking if the latex_code ends with 'end{table}'. This extends the validation to cover table environments that may not strictly end with 'end{tabular}', thus improving the robustnessof table recognition processing. -
myhloli authored
refactor(pdf_extract_kit): optimize image processing and table recognition logicRefactor the image processing logic for OCR and table recognition to ensure consistency and improve performance. Remove redundant initialization of PIL images, unify image cropping logic, and streamline the handling of formula detection results. Also, adjust the table recognition process to improve integration with the updated image processing logic and enhance overall efficiency.
-
icecraft authored
Co-authored-by:shenguanlin <shenguanlin@pjlab.org.cn>
-
- 07 Aug, 2024 2 commits
- 06 Aug, 2024 1 commit
-
-
myhloli authored
-
- 05 Aug, 2024 2 commits
- 04 Aug, 2024 3 commits
-
-
myhloli authored
-
myhloli authored
If 'table-config' is not present in the configuration file, the table recognition feature will default to being disabled to ensure consistent behavior. This change adds a warning log and sets a default configuration for table recognition when the expected config is missing.
-
myhloli authored
Ensure proper formatting of inline equations by adding spaces outside the equation delimitersto prevent markdown from interpreting the equation content as part of a link. This addresses the issue where inline OCR equations appear without the correct markdown formatting.
-
- 02 Aug, 2024 2 commits
