Unverified Commit da048cbf authored by Sidney233's avatar Sidney233 Committed by GitHub
Browse files

Merge branch 'opendatalab:dev' into dev

parents 5685b22b 8a7fec50
......@@ -121,10 +121,14 @@ inference_result: list[PageInferenceResults] = []
{
"category_id": 2,
"poly": [
99.1906967163086, 100.3119125366211,
730.3707885742188, 100.3119125366211,
730.3707885742188, 245.81326293945312,
99.1906967163086, 245.81326293945312
99.1906967163086,
100.3119125366211,
730.3707885742188,
100.3119125366211,
730.3707885742188,
245.81326293945312,
99.1906967163086,
245.81326293945312
],
"score": 0.9999997615814209
}
......@@ -134,6 +138,29 @@ inference_result: list[PageInferenceResults] = []
"height": 2339,
"width": 1654
}
},
{
"layout_dets": [
{
"category_id": 5,
"poly": [
99.13092803955078,
2210.680419921875,
497.3183898925781,
2210.680419921875,
497.3183898925781,
2264.78076171875,
99.13092803955078,
2264.78076171875
],
"score": 0.9999997019767761
}
],
"page_info": {
"page_no": 1,
"height": 2339,
"width": 1654
}
}
]
```
......@@ -264,27 +291,88 @@ inference_result: list[PageInferenceResults] = []
{
"pdf_info": [
{
"preproc_blocks": [...],
"preproc_blocks": [
{
"type": "text",
"bbox": [
52,
61.956024169921875,
294,
82.99800872802734
],
"lines": [
{
"bbox": [
52,
61.956024169921875,
294,
72.0000228881836
],
"spans": [
{
"bbox": [
54.0,
61.956024169921875,
296.2261657714844,
72.0000228881836
],
"content": "dependent on the service headway and the reliability of the departure ",
"type": "text",
"score": 1.0
}
]
}
]
}
],
"layout_bboxes": [
{
"layout_bbox": [52, 61, 294, 731],
"layout_bbox": [
52,
61,
294,
731
],
"layout_label": "V",
"sub_layout": []
}
],
"page_idx": 0,
"page_size": [612.0, 792.0],
"page_size": [
612.0,
792.0
],
"_layout_tree": [],
"images": [],
"tables": [],
"interline_equations": [],
"discarded_blocks": [],
"para_blocks": [
{
"type": "text",
"bbox": [52, 61.956024169921875, 294, 82.99800872802734],
"bbox": [
52,
61.956024169921875,
294,
82.99800872802734
],
"lines": [
{
"bbox": [52, 61.956024169921875, 294, 72.0000228881836],
"bbox": [
52,
61.956024169921875,
294,
72.0000228881836
],
"spans": [
{
"bbox": [54.0, 61.956024169921875, 296.2261657714844, 72.0000228881836],
"content": "dependent on the service headway...",
"bbox": [
54.0,
61.956024169921875,
296.2261657714844,
72.0000228881836
],
"content": "dependent on the service headway and the reliability of the departure ",
"type": "text",
"score": 1.0
}
......@@ -334,38 +422,60 @@ inference_result: list[PageInferenceResults] = []
```json
[
{
{
"type": "text",
"text": "The response of flow duration curves to afforestation",
"text": "The response of flow duration curves to afforestation ",
"text_level": 1,
"page_idx": 0
},
{
"type": "text",
"text": "Abstract",
"type": "text",
"text": "Received 1 October 2003; revised 22 December 2004; accepted 3 January 2005 ",
"page_idx": 0
},
{
"type": "text",
"text": "Abstract ",
"text_level": 2,
"page_idx": 0
},
{
"type": "text",
"text": "The hydrologic effect of replacing pasture or other short crops with trees is reasonably well understood on a mean annual basis. The impact on flow regime, as described by the annual flow duration curve (FDC) is less certain. A method to assess the impact of plantation establishment on FDCs was developed. The starting point for the analyses was the assumption that rainfall and vegetation age are the principal drivers of evapotranspiration. A key objective was to remove the variability in the rainfall signal, leaving changes in streamflow solely attributable to the evapotranspiration of the plantation. A method was developed to (1) fit a model to the observed annual time series of FDC percentiles; i.e. 10th percentile for each year of record with annual rainfall and plantation age as parameters, (2) replace the annual rainfall variation with the long term mean to obtain climate adjusted FDCs, and (3) quantify changes in FDC percentiles as plantations age. Data from 10 catchments from Australia, South Africa and New Zealand were used. The model was able to represent flow variation for the majority of percentiles at eight of the 10 catchments, particularly for the 10–50th percentiles. The adjusted FDCs revealed variable patterns in flow reductions with two types of responses (groups) being identified. Group 1 catchments show a substantial increase in the number of zero flow days, with low flows being more affected than high flows. Group 2 catchments show a more uniform reduction in flows across all percentiles. The differences may be partly explained by storage characteristics. The modelled flow reductions were in accord with published results of paired catchment experiments. An additional analysis was performed to characterise the impact of afforestation on the number of zero flow days $( N _ { \\mathrm { z e r o } } )$ for the catchments in group 1. This model performed particularly well, and when adjusted for climate, indicated a significant increase in $N _ { \\mathrm { z e r o } }$ . The zero flow day method could be used to determine change in the occurrence of any given flow in response to afforestation. The methods used in this study proved satisfactory in removing the rainfall variability, and have added useful insight into the hydrologic impacts of plantation establishment. This approach provides a methodology for understanding catchment response to afforestation, where paired catchment data is not available. ",
"page_idx": 0
},
{
"type": "text",
"text": "1. Introduction ",
"text_level": 2,
"page_idx": 1
},
{
"type": "image",
"img_path": "images/a8ecda1c69b27e4f79fce1589175a9d721cbdc1cf78b4cc06a015f3746f6b9d8.jpg",
"img_caption": ["Fig. 1. Annual flow duration curves..."],
"img_caption": [
"Fig. 1. Annual flow duration curves of daily flows from Pine Creek, Australia, 1989–2000. "
],
"img_footnote": [],
"page_idx": 1
},
{
"type": "equation",
"img_path": "images/181ea56ef185060d04bf4e274685f3e072e922e7b839f093d482c29bf89b71e8.jpg",
"text": "$$Q_{\\%} = f(P) + g(T)$$",
"img_path": "images/181ea56ef185060d04bf4e274685f3e072e922e7b839f093d482c29bf89b71e8.jpg",
"text": "$$\nQ _ { \\% } = f ( P ) + g ( T )\n$$",
"text_format": "latex",
"page_idx": 2
},
{
"type": "table",
"img_path": "images/e3cb413394a475e555807ffdad913435940ec637873d673ee1b039e3bc3496d0.jpg",
"table_caption": ["Table 2 Significance of the rainfall and time terms"],
"table_footnote": ["indicates that the rainfall term was significant..."],
"table_body": "<html><body><table>...</table></body></html>",
"table_caption": [
"Table 2 Significance of the rainfall and time terms "
],
"table_footnote": [
"indicates that the rainfall term was significant at the $5 \\%$ level, $T$ indicates that the time term was significant at the $5 \\%$ level, \\* represents significance at the $10 \\%$ level, and na denotes too few data points for meaningful analysis. "
],
"table_body": "<html><body><table><tr><td rowspan=\"2\">Site</td><td colspan=\"10\">Percentile</td></tr><tr><td>10</td><td>20</td><td>30</td><td>40</td><td>50</td><td>60</td><td>70</td><td>80</td><td>90</td><td>100</td></tr><tr><td>Traralgon Ck</td><td>P</td><td>P,*</td><td>P</td><td>P</td><td>P,</td><td>P,</td><td>P,</td><td>P,</td><td>P</td><td>P</td></tr><tr><td>Redhill</td><td>P,T</td><td>P,T</td><td>,*</td><td>**</td><td>P.T</td><td>P,*</td><td>P*</td><td>P*</td><td>*</td><td>,*</td></tr><tr><td>Pine Ck</td><td></td><td>P,T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>T</td><td>T</td><td>T</td><td>na</td><td>na</td></tr><tr><td>Stewarts Ck 5</td><td>P,T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>P.T</td><td>P.T</td><td>P,T</td><td>na</td><td>na</td><td>na</td></tr><tr><td>Glendhu 2</td><td>P</td><td>P,T</td><td>P,*</td><td>P,T</td><td>P.T</td><td>P,ns</td><td>P,T</td><td>P,T</td><td>P,T</td><td>P,T</td></tr><tr><td>Cathedral Peak 2</td><td>P,T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>*,T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>T</td></tr><tr><td>Cathedral Peak 3</td><td>P.T</td><td>P.T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>T</td></tr><tr><td>Lambrechtsbos A</td><td>P,T</td><td>P</td><td>P</td><td>P,T</td><td>*,T</td><td>*,T</td><td>*,T</td><td>*,T</td><td>*,T</td><td>T</td></tr><tr><td>Lambrechtsbos B</td><td>P,T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>P,T</td><td>T</td><td>T</td></tr><tr><td>Biesievlei</td><td>P,T</td><td>P.T</td><td>P,T</td><td>P,T</td><td>*,T</td><td>*,T</td><td>T</td><td>T</td><td>P,T</td><td>P,T</td></tr></table></body></html>",
"page_idx": 5
}
]
......
......@@ -18,8 +18,8 @@
### 参数传递说明
> [!TIP]
> - 如果您想了解更多有关`sglang`的参数使用方法,请参考 [sglang官方文档](https://docs.sglang.ai/backend/server_arguments.html#common-launch-commands)
> - 所有sglang官方支持的参数都可用通过命令行参数传递给 MinerU,包括以下命令:`mineru`、`mineru-sglang-server`、`mineru-gradio`、`mineru-api`
> - 如果您想了解更多有关`sglang`的参数使用方法,请参考 [sglang官方文档](https://docs.sglang.ai/backend/server_arguments.html#common-launch-commands)
## GPU 设备选择与配置
......@@ -35,23 +35,28 @@
> [!TIP]
> 以下是一些常见的 `CUDA_VISIBLE_DEVICES` 设置示例:
> ```bash
> CUDA_VISIBLE_DEVICES=1 Only device 1 will be seen
> CUDA_VISIBLE_DEVICES=0,1 Devices 0 and 1 will be visible
> CUDA_VISIBLE_DEVICES="0,1" Same as above, quotation marks are optional
> CUDA_VISIBLE_DEVICES=0,2,3 Devices 0, 2, 3 will be visible; device 1 is masked
> CUDA_VISIBLE_DEVICES="" No GPU will be visible
> CUDA_VISIBLE_DEVICES=1 # Only device 1 will be seen
> CUDA_VISIBLE_DEVICES=0,1 # Devices 0 and 1 will be visible
> CUDA_VISIBLE_DEVICES="0,1" # Same as above, quotation marks are optional
> CUDA_VISIBLE_DEVICES=0,2,3 # Devices 0, 2, 3 will be visible; device 1 is masked
> CUDA_VISIBLE_DEVICES="" # No GPU will be visible
> ```
### 实际应用场景
## 实际应用场景
> [!TIP]
> 以下是一些可能的使用场景:
>
> - 如果您有多张显卡,需要指定卡0和卡1,并使用多卡并行来启动'sglang-server',可以使用以下命令:
> - 如果您有多张显卡,需要指定卡0和卡1,并使用多卡并行来启动`sglang-server`,可以使用以下命令:
> ```bash
> CUDA_VISIBLE_DEVICES=0,1 mineru-sglang-server --port 30000 --dp-size 2
> ```
>
> - 如果您有多张显卡,需要指定卡0-3,并使用多卡数据并行和张量并行来启动`sglang-server`,可以使用以下命令:
> ```bash
> CUDA_VISIBLE_DEVICES=0,1,2,3 mineru-sglang-server --port 30000 --dp-size 2 --tp-size 2
> ```
>
> - 如果您有多张显卡,需要在卡0和卡1上启动两个`fastapi`服务,并分别监听不同的端口,可以使用以下命令:
> ```bash
> # 在终端1中
......
......@@ -19,7 +19,7 @@ mineru -p <input_path> -o <output_path>
> - `<input_path>`:本地 PDF/图片 文件或目录
> - `<output_path>`:输出目录
>
> 更多关于输出文件的信息,请参考[输出文件说明](./output_file.md)。
> 更多关于输出文件的信息,请参考[输出文件说明](../output_files.md)。
> [!NOTE]
> 命令行工具会在Linux和macOS系统自动尝试cuda/mps加速。Windows用户如需使用cuda加速,
......@@ -44,7 +44,8 @@ mineru -p <input_path> -o <output_path> -b vlm-transformers
```bash
mineru-api --host 127.0.0.1 --port 8000
```
在浏览器中访问 http://127.0.0.1:8000/docs 查看API文档。
>[!TIP]
>在浏览器中访问 `http://127.0.0.1:8000/docs` 查看API文档。
- 启动gradio webui 可视化前端:
```bash
# 使用 pipeline/vlm-transformers/vlm-sglang-client 后端
......@@ -52,14 +53,21 @@ mineru -p <input_path> -o <output_path> -b vlm-transformers
# 或使用 vlm-sglang-engine/pipeline 后端(需安装sglang环境)
mineru-gradio --server-name 127.0.0.1 --server-port 7860 --enable-sglang-engine true
```
在浏览器中访问 http://127.0.0.1:7860 使用 Gradio WebUI 或访问 http://127.0.0.1:7860/?view=api 使用 Gradio API。
>[!TIP]
>
>- 在浏览器中访问 `http://127.0.0.1:7860` 使用 Gradio WebUI。
>- 访问 `http://127.0.0.1:7860/?view=api` 使用 Gradio API。
- 使用`sglang-client/server`方式调用:
```bash
# 启动sglang server(需要安装sglang环境)
mineru-sglang-server --port 30000
# 在另一个终端中通过sglang client连接sglang server(只需cpu与网络,不需要sglang环境)
mineru -p <input_path> -o <output_path> -b vlm-sglang-client -u http://127.0.0.1:30000
```
>[!TIP]
>在另一个终端中通过sglang client连接sglang server(只需cpu与网络,不需要sglang环境)
> ```bash
> mineru -p <input_path> -o <output_path> -b vlm-sglang-client -u http://127.0.0.1:30000
> ```
> [!TIP]
> 所有sglang官方支持的参数都可用通过命令行参数传递给 MinerU,包括以下命令:`mineru`、`mineru-sglang-server`、`mineru-gradio`、`mineru-api`,
> 我们整理了一些`sglang`使用中的常用参数和使用方法,可以在文档[命令行参数进阶技巧](./advanced_cli_parameters.md)中获取。
......@@ -67,9 +75,10 @@ mineru -p <input_path> -o <output_path> -b vlm-transformers
## 基于配置文件扩展 MinerU 功能
- MinerU 现已实现开箱即用,但也支持通过配置文件扩展功能。您可以在用户目录下创建 `mineru.json` 文件,添加自定义配置。
- `mineru.json` 文件会在您使用内置模型下载命令 `mineru-models-download` 时自动生成,也可以通过将[配置模板文件](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json)复制到用户目录下并重命名为 `mineru.json` 来创建。
- 以下是一些可用的配置选项:
- `latex-delimiter-config`:用于配置 LaTeX 公式的分隔符,默认为`$`符号,可根据需要修改为其他符号或字符串。
- `llm-aided-config`:用于配置 LLM 辅助标题分级的相关参数,兼容所有支持`openai协议`的 LLM 模型,默认使用`阿里云百炼``qwen2.5-32b-instruct`模型,您需要自行配置 API 密钥并将`enable`设置为`true`来启用此功能。
- `models-dir`:用于指定本地模型存储目录,请为`pipeline``vlm`后端分别指定模型目录,指定目录后您可通过配置环境变量`export MINERU_MODEL_SOURCE=local`来使用本地模型。
MinerU 现已实现开箱即用,但也支持通过配置文件扩展功能。您可以在用户目录下创建 `mineru.json` 文件,添加自定义配置。
`mineru.json` 文件会在您使用内置模型下载命令 `mineru-models-download` 时自动生成,也可以通过将[配置模板文件](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json)复制到用户目录下并重命名为 `mineru.json` 来创建。
以下是一些可用的配置选项:
- `latex-delimiter-config`:用于配置 LaTeX 公式的分隔符,默认为`$`符号,可根据需要修改为其他符号或字符串。
- `llm-aided-config`:用于配置 LLM 辅助标题分级的相关参数,兼容所有支持`openai协议`的 LLM 模型,默认使用`阿里云百炼``qwen2.5-32b-instruct`模型,您需要自行配置 API 密钥并将`enable`设置为`true`来启用此功能。
- `models-dir`:用于指定本地模型存储目录,请为`pipeline``vlm`后端分别指定模型目录,指定目录后您可通过配置环境变量`export MINERU_MODEL_SOURCE=local`来使用本地模型。
......@@ -39,6 +39,7 @@ mineru-models-download
```
>[!TIP]
>- 下载完成后,模型路径会在当前终端窗口输出,并自动写入用户目录下的 `mineru.json`。
>- 您也可以通过将[配置模板文件](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json)复制到用户目录下并重命名为 `mineru.json` 来创建配置文件。
>- 模型下载到本地后,您可以自由移动模型文件夹到其他位置,同时需要在 `mineru.json` 中更新模型路径。
>- 如您将模型文件夹部署到其他服务器上,请确保将 `mineru.json`文件一同移动到新设备的用户目录中并正确配置模型路径。
>- 如您需要更新模型文件,可以再次运行 `mineru-models-download` 命令,模型更新暂不支持自定义路径,如您没有移动本地模型文件夹,模型文件会增量更新;如您移动了模型文件夹,模型文件会重新下载到默认位置并更新`mineru.json`。
......
......@@ -30,35 +30,49 @@ theme:
icon: material/brightness-4
name: Switch to system preference
logo: images/logo.png
favicon: images/logo.svg
favicon: images/logo.png
features:
- content.tabs.link
- content.code.annotate
- content.code.copy
- navigation.footer
- navigation.tabs
- navigation.instant
- navigation.instant.prefetch
- navigation.instant.progress
- navigation.tabs
- navigation.tabs.sticky
- navigation.sections
- navigation.path
- navigation.indexes
- navigation.top
- navigation.tracking
- search.suggest
- toc.follow
nav:
- Home:
- "MinerU": index.md
- Quick Start:
- quick_start/index.md
- Quick Start: quick_start/index.md
- Extension Modules: quick_start/extension_modules.md
- Docker Deployment: quick_start/docker_deployment.md
- Usage:
- usage/index.md
- Usage: usage/index.md
- CLI Tools: usage/cli_tools.md
- Model Source: usage/model_source.md
- Advanced CLI Parameters: usage/advanced_cli_parameters.md
- Output File Format: usage/output_files.md
- Reference:
- Output File Format: reference/output_files.md
- FAQ:
- FAQ: faq/index.md
- Demo:
- Demo: demo/index.md
- Reference:
- Output File Format: reference/output_files.md
- FAQ:
- FAQ: FAQ/index.md
- FAQ: faq/index.md
- Demo:
- Demo: demo/index.md
plugins:
- search
......@@ -75,17 +89,21 @@ plugins:
nav_translations:
Home: 主页
Quick Start: 快速开始
Extension Modules: 扩展模块
Extension Modules: 扩展模块安装
Docker Deployment: Docker部署
Usage: 使用方法
CLI Tools: 命令行工具
Model Source: 模型源
Advanced CLI Parameters: 命令行参数进阶技巧
FAQ: FAQ
Advanced CLI Parameters: 命令行进阶参数
FAQ: 常见问题解答
Reference: 参考资料
Output File Format: 输出文件格式
- mkdocs-video
markdown_extensions:
- admonition
- pymdownx.details
- attr_list
- gfm_admonition
- pymdownx.highlight:
use_pygments: true
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment