Skip to content

GitLab

"...resnet50_tensorflow.git" did not exist on "be8e93fe73f011f2bff5ebb7caf877282fdb86f1"
Commit dd8da7bf authored by myhloli's avatar myhloli
Browse files

Merge remote-tracking branch 'origin/dev' into dev

parents 8ea23813 74fba476
Hide whitespace changes
Inline Side-by-side
Showing with 126 additions and 514 deletions
.gitignore
View file @ dd8da7bf
......@@ -48,3 +48,6 @@ debug_utils/
# sphinx docs
_build/
output/
\ No newline at end of file
README.md
View file @ dd8da7bf
......@@ -75,12 +75,10 @@
<ul>
<li><a href="#online-demo">Online Demo</a></li>
<li><a href="#quick-cpu-demo">Quick CPU Demo</a></li>
<li><a href="#using-gpu">Using GPU</a></li>
</ul>
</li>
<li><a href="#usage">Usage</a>
<ul>
<li><a href="#command-line">Command Line</a></li>
<li><a href="#api">API</a></li>
<li><a href="#deploy-derived-projects">Deploy Derived Projects</a></li>
<li><a href="#development-guide">Development Guide</a></li>
......@@ -89,8 +87,6 @@
</ul>
</li>
<li><a href="#todo">TODO</a></li>
<li><a href="#known-issues">Known Issues</a></li>
<li><a href="#faq">FAQ</a></li>
<li><a href="#all-thanks-to-our-contributors">All Thanks To Our Contributors</a></li>
<li><a href="#license-information">License Information</a></li>
<li><a href="#acknowledgments">Acknowledgments</a></li>
......@@ -112,89 +108,12 @@ Compared to well-known commercial products, MinerU is still young. If you encoun
https://github.com/user-attachments/assets/4bea02c9-6d54-4cd6-97ed-dff14340982c
## Key Features
- Remove headers, footers, footnotes, page numbers, etc., to ensure semantic coherence.
- Output text in human-readable order, suitable for single-column, multi-column, and complex layouts.
- Preserve the structure of the original document, including headings, paragraphs, lists, etc.
- Extract images, image descriptions, tables, table titles, and footnotes.
- Automatically recognize and convert formulas in the document to LaTeX format.
- Automatically recognize and convert tables in the document to LaTeX or HTML format.
- Automatically detect scanned PDFs and garbled PDFs and enable OCR functionality.
- OCR supports detection and recognition of 84 languages.
- Supports multiple output formats, such as multimodal and NLP Markdown, JSON sorted by reading order, and rich intermediate formats.
- Supports various visualization results, including layout visualization and span visualization, for efficient confirmation of output quality.
- Supports both CPU and GPU environments.
- Compatible with Windows, Linux, and Mac platforms.
## Quick Start
If you encounter any installation issues, please first consult the <a href="#faq">FAQ</a>. </br>
If the parsing results are not as expected, refer to the <a href="#known-issues">Known Issues</a>. </br>
There are three different ways to experience MinerU:
There are multiple different ways to experience MinerU:
- [Online Demo (No Installation Required)](#online-demo)
- [Quick CPU Demo (Windows, Linux, Mac)](#quick-cpu-demo)
- [Linux/Windows + CUDA](#Using-GPU)
> [!WARNING]
> **Pre-installation Notice—Hardware and Software Environment Support**
>
> To ensure the stability and reliability of the project, we only optimize and test for specific hardware and software environments during development. This ensures that users deploying and running the project on recommended system configurations will get the best performance with the fewest compatibility issues.
>
> By focusing resources on the mainline environment, our team can more efficiently resolve potential bugs and develop new features.
>
> In non-mainline environments, due to the diversity of hardware and software configurations, as well as third-party dependency compatibility issues, we cannot guarantee 100% project availability. Therefore, for users who wish to use this project in non-recommended environments, we suggest carefully reading the documentation and FAQ first. Most issues already have corresponding solutions in the FAQ. We also encourage community feedback to help us gradually expand support.
<table>
<tr>
<td colspan="3" rowspan="2">Operating System</td>
</tr>
<tr>
<td>Ubuntu 22.04 LTS</td>
<td>Windows 10 / 11</td>
<td>macOS 11+</td>
</tr>
<tr>
<td colspan="3">CPU</td>
<td>x86_64(unsupported ARM Linux)</td>
<td>x86_64(unsupported ARM Windows)</td>
<td>x86_64 / arm64</td>
</tr>
<tr>
<td colspan="3">Memory</td>
<td colspan="3">16GB or more, recommended 32GB+</td>
</tr>
<tr>
<td colspan="3">Python Version</td>
<td colspan="3">3.10(Please make sure to create a Python 3.10 virtual environment using conda)</td>
</tr>
<tr>
<td colspan="3">Nvidia Driver Version</td>
<td>latest (Proprietary Driver)</td>
<td>latest</td>
<td>None</td>
</tr>
<tr>
<td colspan="3">CUDA Environment</td>
<td>Automatic installation [12.1 (pytorch) + 11.8 (paddle)]</td>
<td>11.8 (manual installation) + cuDNN v8.7.0 (manual installation)</td>
<td>None</td>
</tr>
<tr>
<td rowspan="2">GPU Hardware Support List</td>
<td colspan="2">Minimum Requirement 8G+ VRAM</td>
<td colspan="2">3060ti/3070/4060<br>
8G VRAM enables layout, formula recognition acceleration and OCR acceleration</td>
<td rowspan="2">None</td>
</tr>
<tr>
<td colspan="2">Recommended Configuration 10G+ VRAM</td>
<td colspan="2">3080/3080ti/3090/3090ti/4070/4070ti/4070tisuper/4080/4090<br>
10G VRAM or more can enable layout, formula recognition, OCR acceleration and table recognition acceleration simultaneously
</td>
</tr>
</table>
### Online Demo
......@@ -251,85 +170,8 @@ You can modify certain configurations in this file to enable or disable features
}
```
### Using GPU
If your device supports CUDA and meets the GPU requirements of the mainline environment, you can use GPU acceleration. Please select the appropriate guide based on your system:
- [Ubuntu 22.04 LTS + GPU](docs/README_Ubuntu_CUDA_Acceleration_en_US.md)
- [Windows 10/11 + GPU](docs/README_Windows_CUDA_Acceleration_en_US.md)
- Quick Deployment with Docker
> [!IMPORTANT]
> Docker requires a GPU with at least 16GB of VRAM, and all acceleration features are enabled by default.
>
> Before running this Docker, you can use the following command to check if your device supports CUDA acceleration on Docker.
>
> ```bash
> docker run --rm --gpus=all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
> ```
```bash
wget https://github.com/opendatalab/MinerU/raw/master/Dockerfile
docker build -t mineru:latest .
docker run --rm -it --gpus=all mineru:latest /bin/bash
magic-pdf --help
```
## Usage
### Command Line
```bash
magic-pdf --help
Usage: magic-pdf [OPTIONS]
Options:
-v, --version display the version and exit
-p, --path PATH local pdf filepath or directory [required]
-o, --output-dir PATH output local directory [required]
-m, --method [ocr|txt|auto] the method for parsing pdf. ocr: using ocr
technique to extract information from pdf. txt:
suitable for the text-based pdf only and
outperform ocr. auto: automatically choose the
best method for parsing pdf from ocr and txt.
without method specified, auto will be used by
default.
-l, --lang TEXT Input the languages in the pdf (if known) to
improve OCR accuracy. Optional. You should
input "Abbreviation" with language form url: ht
tps://paddlepaddle.github.io/PaddleOCR/latest/en
/ppocr/blog/multi_languages.html#5-support-languages-
and-abbreviations
-d, --debug BOOLEAN Enables detailed debugging information during
the execution of the CLI commands.
-s, --start INTEGER The starting page for PDF parsing, beginning
from 0.
-e, --end INTEGER The ending page for PDF parsing, beginning from
0.
--help Show this message and exit.
## show version
magic-pdf -v
## command line example
magic-pdf -p {some_pdf} -o {some_output_dir} -m auto
```
`{some_pdf}` can be a single PDF file or a directory containing multiple PDFs.
The results will be saved in the `{some_output_dir}` directory. The output file list is as follows:
```text
├── some_pdf.md # markdown file
├── images # directory for storing images
├── some_pdf_layout.pdf # layout diagram (Include layout reading order)
├── some_pdf_middle.json # MinerU intermediate processing result
├── some_pdf_model.json # model inference result
├── some_pdf_origin.pdf # original PDF file
├── some_pdf_spans.pdf # smallest granularity bbox position information diagram
└── some_pdf_content_list.json # Rich text JSON arranged in reading order
```
> [!TIP]
> For more information about the output files, please refer to the [Output File Description](docs/output_file_en_us.md).
### API
Processing files from local disk
......@@ -386,24 +228,6 @@ TODO
- [ ] [Chemical formula recognition](docs/chemical_knowledge_introduction/introduction.pdf)
- [ ] Geometric shape recognition
# Known Issues
- Reading order is determined by the model based on the spatial distribution of readable content, and may be out of order in some areas under extremely complex layouts.
- Vertical text is not supported.
- Tables of contents and lists are recognized through rules, and some uncommon list formats may not be recognized.
- Only one level of headings is supported; hierarchical headings are not currently supported.
- Code blocks are not yet supported in the layout model.
- Comic books, art albums, primary school textbooks, and exercises cannot be parsed well.
- Table recognition may result in row/column recognition errors in complex tables.
- OCR recognition may produce inaccurate characters in PDFs of lesser-known languages (e.g., diacritical marks in Latin script, easily confused characters in Arabic script).
- Some formulas may not render correctly in Markdown.
# FAQ
[FAQ in Chinese](docs/FAQ_zh_cn.md)
[FAQ in English](docs/FAQ_en_us.md)
# All Thanks To Our Contributors
<a href="https://github.com/opendatalab/MinerU/graphs/contributors">
......
README_zh-CN.md
View file @ dd8da7bf
......@@ -76,12 +76,10 @@
<ul>
<li><a href="#在线体验">在线体验</a></li>
<li><a href="#使用CPU快速体验">使用CPU快速体验</a></li>
<li><a href="#使用GPU">使用GPU</a></li>
</ul>
</li>
<li><a href="#使用">使用方式</a>
<ul>
<li><a href="#命令行">命令行</a></li>
<li><a href="#api">API</a></li>
<li><a href="#部署衍生项目">部署衍生项目</a></li>
<li><a href="#二次开发">二次开发</a></li>
......@@ -113,90 +111,13 @@ MinerU诞生于[书生-浦语](https://github.com/InternLM/InternLM)的预训练
https://github.com/user-attachments/assets/4bea02c9-6d54-4cd6-97ed-dff14340982c
## 主要功能
- 删除页眉、页脚、脚注、页码等元素，确保语义连贯
- 输出符合人类阅读顺序的文本，适用于单栏、多栏及复杂排版
- 保留原文档的结构，包括标题、段落、列表等
- 提取图像、图片描述、表格、表格标题及脚注
- 自动识别并转换文档中的公式为LaTeX格式
- 自动识别并转换文档中的表格为LaTeX或HTML格式
- 自动检测扫描版PDF和乱码PDF，并启用OCR功能
- OCR支持84种语言的检测与识别
- 支持多种输出格式，如多模态与NLP的Markdown、按阅读顺序排序的JSON、含有丰富信息的中间格式等
- 支持多种可视化结果，包括layout可视化、span可视化等，便于高效确认输出效果与质检
- 支持CPU和GPU环境
- 兼容Windows、Linux和Mac平台
## 快速开始
如果遇到任何安装问题，请先查询 <a href="#faq">FAQ</a> </br>
如果遇到解析效果不及预期，参考 <a href="#known-issues">Known Issues</a></br>
有3种不同方式可以体验MinerU的效果：
有多种不同方式可以体验MinerU的效果：
- [在线体验(无需任何安装)](#在线体验)
- [使用CPU快速体验（Windows，Linux，Mac）](#使用cpu快速体验)
- [Linux/Windows + CUDA](#使用gpu)
> [!WARNING]
> **安装前必看——软硬件环境支持说明**
>
> 为了确保项目的稳定性和可靠性，我们在开发过程中仅对特定的软硬件环境进行优化和测试。这样当用户在推荐的系统配置上部署和运行项目时，能够获得最佳的性能表现和最少的兼容性问题。
>
> 通过集中资源和精力于主线环境，我们团队能够更高效地解决潜在的BUG，及时开发新功能。
>
> 在非主线环境中，由于硬件、软件配置的多样性，以及第三方依赖项的兼容性问题，我们无法100%保证项目的完全可用性。因此，对于希望在非推荐环境中使用本项目的用户，我们建议先仔细阅读文档以及FAQ，大多数问题已经在FAQ中有对应的解决方案，除此之外我们鼓励社区反馈问题，以便我们能够逐步扩大支持范围。
<table>
<tr>
<td colspan="3" rowspan="2">操作系统</td>
</tr>
<tr>
<td>Ubuntu 22.04 LTS</td>
<td>Windows 10 / 11</td>
<td>macOS 11+</td>
</tr>
<tr>
<td colspan="3">CPU</td>
<td>x86_64(暂不支持ARM Linux)</td>
<td>x86_64(暂不支持ARM Windows)</td>
<td>x86_64 / arm64</td>
</tr>
<tr>
<td colspan="3">内存</td>
<td colspan="3">大于等于16GB，推荐32G以上</td>
</tr>
<tr>
<td colspan="3">python版本</td>
<td colspan="3">3.10 (请务必通过conda创建3.10虚拟环境)</td>
</tr>
<tr>
<td colspan="3">Nvidia Driver 版本</td>
<td>latest(专有驱动)</td>
<td>latest</td>
<td>None</td>
</tr>
<tr>
<td colspan="3">CUDA环境</td>
<td>自动安装[12.1(pytorch)+11.8(paddle)]</td>
<td>11.8(手动安装)+cuDNN v8.7.0(手动安装)</td>
<td>None</td>
</tr>
<tr>
<td rowspan="2">GPU硬件支持列表</td>
<td colspan="2">最低要求 8G+显存</td>
<td colspan="2">3060ti/3070/4060<br>
8G显存可开启layout、公式识别和ocr加速</td>
<td rowspan="2">None</td>
</tr>
<tr>
<td colspan="2">推荐配置 10G+显存</td>
<td colspan="2">3080/3080ti/3090/3090ti/4070/4070ti/4070tisuper/4080/4090<br>
10G显存及以上可以同时开启layout、公式识别和ocr加速和表格识别加速<br>
</td>
</tr>
</table>
### 在线体验
稳定版(经过QA验证的稳定版本)：
......@@ -257,87 +178,9 @@ pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com -i h
}
```
### 使用GPU
如果您的设备支持CUDA，且满足主线环境中的显卡要求，则可以使用GPU加速，请根据自己的系统选择适合的教程：
- [Ubuntu22.04LTS + GPU](docs/README_Ubuntu_CUDA_Acceleration_zh_CN.md)
- [Windows10/11 + GPU](docs/README_Windows_CUDA_Acceleration_zh_CN.md)
- 使用Docker快速部署
> [!IMPORTANT]
> Docker 需设备gpu显存大于等于16GB，默认开启所有加速功能
>
> 运行本docker前可以通过以下命令检测自己的设备是否支持在docker上使用CUDA加速
>
> ```bash
> docker run --rm --gpus=all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
> ```
```bash
wget https://github.com/opendatalab/MinerU/raw/master/Dockerfile
docker build -t mineru:latest .
docker run --rm -it --gpus=all mineru:latest /bin/bash
magic-pdf --help
```
## 使用
### 命令行
```bash
magic-pdf --help
Usage: magic-pdf [OPTIONS]
Options:
-v, --version display the version and exit
-p, --path PATH local pdf filepath or directory [required]
-o, --output-dir PATH output local directory [required]
-m, --method [ocr|txt|auto] the method for parsing pdf. ocr: using ocr
technique to extract information from pdf. txt:
suitable for the text-based pdf only and
outperform ocr. auto: automatically choose the
best method for parsing pdf from ocr and txt.
without method specified, auto will be used by
default.
-l, --lang TEXT Input the languages in the pdf (if known) to
improve OCR accuracy. Optional. You should
input "Abbreviation" with language form url: ht
tps://paddlepaddle.github.io/PaddleOCR/latest/en
/ppocr/blog/multi_languages.html#5-support-languages-
and-abbreviations
-d, --debug BOOLEAN Enables detailed debugging information during
the execution of the CLI commands.
-s, --start INTEGER The starting page for PDF parsing, beginning
from 0.
-e, --end INTEGER The ending page for PDF parsing, beginning from
0.
--help Show this message and exit.
## show version
magic-pdf -v
## command line example
magic-pdf -p {some_pdf} -o {some_output_dir} -m auto
```
其中 `{some_pdf}` 可以是单个pdf文件，也可以是一个包含多个pdf文件的目录。
运行完命令后输出的结果会保存在`{some_output_dir}`目录下, 输出的文件列表如下
```text
├── some_pdf.md # markdown 文件
├── images # 存放图片目录
├── some_pdf_layout.pdf # layout 绘图 （包含layout阅读顺序）
├── some_pdf_middle.json # minerU 中间处理结果
├── some_pdf_model.json # 模型推理结果
├── some_pdf_origin.pdf # 原 pdf 文件
├── some_pdf_spans.pdf # 最小粒度的bbox位置信息绘图
└── some_pdf_content_list.json # 按阅读顺序排列的富文本json
```
> [!TIP]
> 更多有关输出文件的信息，请参考[输出文件说明](docs/output_file_zh_cn.md)
### API
处理本地磁盘上的文件
......@@ -394,24 +237,6 @@ TODO
- [ ] [化学式识别](docs/chemical_knowledge_introduction/introduction.pdf)
- [ ] 几何图形识别
# Known Issues
- 阅读顺序基于模型对可阅读内容在空间中的分布进行排序，在极端复杂的排版下可能会部分区域乱序
- 不支持竖排文字
- 目录和列表通过规则进行识别，少部分不常见的列表形式可能无法识别
- 标题只有一级，目前不支持标题分级
- 代码块在layout模型里还没有支持
- 漫画书、艺术图册、小学教材、习题尚不能很好解析
- 表格识别在复杂表格上可能会出现行/列识别错误
- 在小语种PDF上，OCR识别可能会出现字符不准确的情况（如拉丁文的重音符号、阿拉伯文易混淆字符等）
- 部分公式可能会无法在markdown中渲染
# FAQ
[常见问题](docs/FAQ_zh_cn.md)
[FAQ](docs/FAQ_en_us.md)
# All Thanks To Our Contributors
......
next_docs/en/additional_notes/changelog.rst deleted 100644 → 0
View file @ 8ea23813
Changelog
=========
- 2024/09/27 Version 0.8.1 released, Fixed some bugs, and providing a
`localized deployment version <projects/web_demo/README.md>`__ of the
`online
demo <https://opendatalab.com/OpenSourceTools/Extractor/PDF/>`__ and
the `front-end interface <projects/web/README.md>`__.
- 2024/09/09: Version 0.8.0 released, supporting fast deployment with
Dockerfile, and launching demos on Huggingface and Modelscope.
- 2024/08/30: Version 0.7.1 released, add paddle tablemaster table
recognition option
- 2024/08/09: Version 0.7.0b1 released, simplified installation
process, added table recognition functionality
- 2024/08/01: Version 0.6.2b1 released, optimized dependency conflict
issues and installation documentation
- 2024/07/05: Initial open-source release
.. warning::
fix ``localized deployment version`` and ``front-end interface``
next_docs/en/additional_notes/faq.rst
View file @ dd8da7bf
......@@ -74,3 +74,15 @@ CUDA version used by Paddle needs to be upgraded.
pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/
Reference: https://github.com/opendatalab/MinerU/issues/558
7. On some Linux servers, the program immediately reports an error ``Illegal instruction (core dumped)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This might be because the server's CPU does not support the AVX/AVX2
instruction set, or the CPU itself supports it but has been disabled by
the system administrator. You can try contacting the system
administrator to remove the restriction or change to a different server.
References: https://github.com/opendatalab/MinerU/issues/591 ,
https://github.com/opendatalab/MinerU/issues/736
next_docs/en/additional_notes/known_issues.rst
View file @ dd8da7bf
Known Issues
============
- Reading order is based on the model’s sorting of text distribution in
space, which may become disordered under extremely complex layouts.
- Reading order is determined by the model based on the spatial
distribution of readable content, and may be out of order in some
areas under extremely complex layouts.
- Vertical text is not supported.
- Tables of contents and lists are recognized through rules; a few
uncommon list formats may not be identified.
- Only one level of headings is supported; hierarchical heading levels
are currently not supported.
- Tables of contents and lists are recognized through rules, and some
uncommon list formats may not be recognized.
- Only one level of headings is supported; hierarchical headings are
not currently supported.
- Code blocks are not yet supported in the layout model.
- Comic books, art books, elementary school textbooks, and exercise
books are not well-parsed yet
- Enabling OCR may produce better results in PDFs with a high density
of formulas
- If you are processing PDFs with a large number of formulas, it is
strongly recommended to enable the OCR function. When using PyMuPDF
to extract text, overlapping text lines can occur, leading to
inaccurate formula insertion positions.
- Comic books, art albums, primary school textbooks, and exercises
cannot be parsed well.
- Table recognition may result in row/column recognition errors in
complex tables.
- OCR recognition may produce inaccurate characters in PDFs of
lesser-known languages (e.g., diacritical marks in Latin script,
easily confused characters in Arabic script).
- Some formulas may not render correctly in Markdown.
\ No newline at end of file
next_docs/en/conf.py
View file @ dd8da7bf
......@@ -95,7 +95,7 @@ language = 'en'
html_theme = 'sphinx_book_theme'
html_logo = '_static/image/logo.png'
html_theme_options = {
'path_to_docs': 'docs/en',
'path_to_docs': 'next_docs/en',
'repository_url': 'https://github.com/opendatalab/MinerU',
'use_repository_button': True,
}
......
next_docs/en/index.rst
View file @ dd8da7bf
......@@ -46,20 +46,29 @@ the relevant PDF**.
Key Features
------------
- Removes elements such as headers, footers, footnotes, and page
numbers while maintaining semantic continuity
- Outputs text in a human-readable order from multi-column documents
- Retains the original structure of the document, including titles,
paragraphs, and lists
- Extracts images, image captions, tables, and table captions
- Automatically recognizes formulas in the document and converts them
to LaTeX
- Automatically recognizes tables in the document and converts them to
LaTeX
- Automatically detects and enables OCR for corrupted PDFs
- Supports both CPU and GPU environments
- Supports Windows, Linux, and Mac platforms
- Remove headers, footers, footnotes, page numbers, etc., to ensure
semantic coherence.
- Output text in human-readable order, suitable for single-column,
multi-column, and complex layouts.
- Preserve the structure of the original document, including headings,
paragraphs, lists, etc.
- Extract images, image descriptions, tables, table titles, and
footnotes.
- Automatically recognize and convert formulas in the document to LaTeX
format.
- Automatically recognize and convert tables in the document to LaTeX
or HTML format.
- Automatically detect scanned PDFs and garbled PDFs and enable OCR
functionality.
- OCR supports detection and recognition of 84 languages.
- Supports multiple output formats, such as multimodal and NLP
Markdown, JSON sorted by reading order, and rich intermediate
formats.
- Supports various visualization results, including layout
visualization and span visualization, for efficient confirmation of
output quality.
- Supports both CPU and GPU environments.
- Compatible with Windows, Linux, and Mac platforms.
User Guide
-------------
......@@ -91,14 +100,6 @@ Additional Notes
additional_notes/known_issues
additional_notes/faq
additional_notes/changelog
additional_notes/glossary
Projects
---------
.. toctree::
:maxdepth: 1
:caption: Projects
projects
\ No newline at end of file
next_docs/en/projects.rst deleted 100644 → 0
View file @ 8ea23813
llama_index_rag
===============
gradio_app
============
other projects
===============
\ No newline at end of file
next_docs/en/user_guide/install/boost_with_cuda.rst
View file @ dd8da7bf
......@@ -137,7 +137,7 @@ Download a sample file from the repository and test it.
.. code:: sh
wget https://github.com/opendatalab/MinerU/raw/master/demo/small_ocr.pdf
magic-pdf -p small_ocr.pdf
magic-pdf -p small_ocr.pdf -o ./output
9. Test CUDA Acceleration
~~~~~~~~~~~~~~~~~~~~~~~~~
......@@ -145,10 +145,6 @@ Download a sample file from the repository and test it.
If your graphics card has at least **8GB** of VRAM, follow these steps
to test CUDA acceleration:
❗ Due to the extremely limited nature of 8GB VRAM for running this
application, you need to close all other programs using VRAM to
ensure that 8GB of VRAM is available when running this application.
1. Modify the value of ``"device-mode"`` in the ``magic-pdf.json``
configuration file located in your home directory.
......@@ -162,7 +158,7 @@ to test CUDA acceleration:
.. code:: sh
magic-pdf -p small_ocr.pdf
magic-pdf -p small_ocr.pdf -o ./output
10. Enable CUDA Acceleration for OCR
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
......@@ -178,7 +174,9 @@ to test CUDA acceleration:
.. code:: sh
magic-pdf -p small_ocr.pdf
magic-pdf -p small_ocr.pdf -o ./output
.. _windows_10_or_11_section:
......@@ -252,7 +250,7 @@ Download a sample file from the repository and test it.
.. code:: powershell
wget https://github.com/opendatalab/MinerU/raw/master/demo/small_ocr.pdf -O small_ocr.pdf
magic-pdf -p small_ocr.pdf
magic-pdf -p small_ocr.pdf -o ./output
8. Test CUDA Acceleration
~~~~~~~~~~~~~~~~~~~~~~~~~
......@@ -260,10 +258,6 @@ Download a sample file from the repository and test it.
If your graphics card has at least 8GB of VRAM, follow these steps to
test CUDA-accelerated parsing performance.
❗ Due to the extremely limited nature of 8GB VRAM for running this
application, you need to close all other programs using VRAM to
ensure that 8GB of VRAM is available when running this application.
1. **Overwrite the installation of torch and torchvision** supporting
CUDA.
......@@ -295,7 +289,7 @@ test CUDA-accelerated parsing performance.
::
magic-pdf -p small_ocr.pdf
magic-pdf -p small_ocr.pdf -o ./output
9. Enable CUDA Acceleration for OCR
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
......@@ -311,5 +305,4 @@ test CUDA-accelerated parsing performance.
::
magic-pdf -p small_ocr.pdf
magic-pdf -p small_ocr.pdf -o ./output
next_docs/en/user_guide/install/install.rst
View file @ dd8da7bf
Install
===============================================================
If you encounter any installation issues, please first consult the FAQ.
If the parsing results are not as expected, refer to the Known Issues.
There are three different ways to experience MinerU
If you encounter any installation issues, please first consult the :doc:`../../additional_notes/faq`.
If the parsing results are not as expected, refer to the :doc:`../../additional_notes/known_issues`.
Pre-installation Notice—Hardware and Software Environment Support
------------------------------------------------------------------
......@@ -27,61 +26,62 @@ community feedback to help us gradually expand support.
.. raw:: html
<style>
table, th, td {
border: 1px solid black;
border-collapse: collapse;
}
</style>
<table>
<tr>
<td colspan="3" rowspan="2">Operating System</td>
</tr>
<tr>
<td>Ubuntu 22.04 LTS</td>
<td>Windows 10 / 11</td>
<td>macOS 11+</td>
</tr>
<tr>
<td colspan="3">CPU</td>
<td>x86_64</td>
<td>x86_64</td>
<td>x86_64 / arm64</td>
</tr>
<tr>
<td colspan="3">Memory</td>
<td colspan="3">16GB or more, recommended 32GB+</td>
</tr>
<tr>
<td colspan="3">Python Version</td>
<td colspan="3">3.10</td>
</tr>
<tr>
<td colspan="3">Nvidia Driver Version</td>
<td>latest (Proprietary Driver)</td>
<td>latest</td>
<td>None</td>
</tr>
<tr>
<td colspan="3">CUDA Environment</td>
<td>Automatic installation [12.1 (pytorch) + 11.8 (paddle)]</td>
<td>11.8 (manual installation) + cuDNN v8.7.0 (manual installation)</td>
<td>None</td>
</tr>
<tr>
<td rowspan="2">GPU Hardware Support List</td>
<td colspan="2">Minimum Requirement 8G+ VRAM</td>
<td colspan="2">3060ti/3070/3080/3080ti/4060/4070/4070ti<br>
8G VRAM enables layout, formula recognition acceleration and OCR acceleration</td>
<td rowspan="2">None</td>
</tr>
<tr>
<td colspan="2">Recommended Configuration 16G+ VRAM</td>
<td colspan="2">3090/3090ti/4070ti super/4080/4090<br>
16G VRAM or more can enable layout, formula recognition, OCR acceleration and table recognition acceleration simultaneously
</td>
</tr>
</table>
<style>
table, th, td {
border: 1px solid black;
border-collapse: collapse;
}
</style>
<table>
<tr>
<td colspan="3" rowspan="2">Operating System</td>
</tr>
<tr>
<td>Ubuntu 22.04 LTS</td>
<td>Windows 10 / 11</td>
<td>macOS 11+</td>
</tr>
<tr>
<td colspan="3">CPU</td>
<td>x86_64(unsupported ARM Linux)</td>
<td>x86_64(unsupported ARM Windows)</td>
<td>x86_64 / arm64</td>
</tr>
<tr>
<td colspan="3">Memory</td>
<td colspan="3">16GB or more, recommended 32GB+</td>
</tr>
<tr>
<td colspan="3">Python Version</td>
<td colspan="3">3.10(Please make sure to create a Python 3.10 virtual environment using conda)</td>
</tr>
<tr>
<td colspan="3">Nvidia Driver Version</td>
<td>latest (Proprietary Driver)</td>
<td>latest</td>
<td>None</td>
</tr>
<tr>
<td colspan="3">CUDA Environment</td>
<td>Automatic installation [12.1 (pytorch) + 11.8 (paddle)]</td>
<td>11.8 (manual installation) + cuDNN v8.7.0 (manual installation)</td>
<td>None</td>
</tr>
<tr>
<td rowspan="2">GPU Hardware Support List</td>
<td colspan="2">Minimum Requirement 8G+ VRAM</td>
<td colspan="2">3060ti/3070/4060<br>
8G VRAM enables layout, formula recognition acceleration and OCR acceleration</td>
<td rowspan="2">None</td>
</tr>
<tr>
<td colspan="2">Recommended Configuration 10G+ VRAM</td>
<td colspan="2">3080/3080ti/3090/3090ti/4070/4070ti/4070tisuper/4080/4090<br>
10G VRAM or more can enable layout, formula recognition, OCR acceleration and table recognition acceleration simultaneously
</td>
</tr>
</table>
Create an environment
......
next_docs/en/user_guide/quick_start/command_line.rst
View file @ dd8da7bf
......@@ -55,5 +55,8 @@ directory. The output file list is as follows:
├── some_pdf_spans.pdf # smallest granularity bbox position information diagram
└── some_pdf_content_list.json # Rich text JSON arranged in reading order
For more information about the output files, please refer to the :doc:`../tutorial/output_file_description`
.. admonition:: Tip
:class: tip
For more information about the output files, please refer to the :doc:`../tutorial/output_file_description`
next_docs/en/user_guide/quick_start/extract_text.rst deleted 100644 → 0
View file @ 8ea23813
Extract Content from Pdf
========================
.. code:: python
from magic_pdf.data.read_api import read_local_pdfs
from magic_pdf.pdf_parse_union_core_v2 import pdf_parse_union
from magic_pdf.model.doc_analyze_by_custom_model import doc_analyze
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment