Merge pull request #2858 from myhloli/dev

docs: update output file documentation for mineru command and enhance content descriptions in output_file_en_us.md and output_file_zh_cn.md

docs/FAQ_en_us.md

 # Frequently Asked Questions
 
-### 1. When using the command `pip install magic-pdf[full]` on newer versions of macOS, the error `zsh: no matches found: magic-pdf[full]` occurs.
-
-On macOS, the default shell has switched from Bash to Z shell, which has special handling logic for certain types of string matching. This can lead to the "no matches found" error. You can try disabling the globbing feature in the command line and then run the installation command again.
-
-```bash
-setopt no_nomatch
-pip install magic-pdf[full]
-```
-
-### 2. Encountering the error `pickle.UnpicklingError: invalid load key, 'v'.` during use
-
-This might be due to an incomplete download of the model file. You can try re-downloading the model file and then try again.
-Reference: https://github.com/opendatalab/MinerU/issues/143
-
-### 3. Where should the model files be downloaded and how should the `/models-dir` configuration be set?
-
-The path for the model files is configured in "magic-pdf.json". just like:
-
-```json
-{
-  "models-dir": "/tmp/models"
-}
-```
-
-This path is an absolute path, not a relative path. You can obtain the absolute path in the models directory using the "pwd" command.
-Reference: https://github.com/opendatalab/MinerU/issues/155#issuecomment-2230216874
-
-### 4. Encountered the error `ImportError: libGL.so.1: cannot open shared object file: No such file or directory` in Ubuntu 22.04 on WSL2
+### 1. Encountered the error `ImportError: libGL.so.1: cannot open shared object file: No such file or directory` in Ubuntu 22.04 on WSL2
 
 The `libgl` library is missing in Ubuntu 22.04 on WSL2. You can install the `libgl` library with the following command to resolve the issue:
 
@@ -37,59 +10,14 @@ sudo apt-get install libgl1-mesa-glx
 
 Reference: https://github.com/opendatalab/MinerU/issues/388
 
-### 5. Encountered error `ModuleNotFoundError: No module named 'fairscale'`
-
-You need to uninstall the module and reinstall it:
-
-```bash
-pip uninstall fairscale
-pip install fairscale
-```
-
-Reference: https://github.com/opendatalab/MinerU/issues/411
-
-### 6. On some newer devices like the H100, the text parsed during OCR using CUDA acceleration is garbled.
 
-The compatibility of cuda11 with new graphics cards is poor, and the CUDA version used by Paddle needs to be upgraded.
-
-```bash
-pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/
-```
-
-Reference: https://github.com/opendatalab/MinerU/issues/558
-
-### 7. On some Linux servers, the program immediately reports an error `Illegal instruction (core dumped)`
-
-This might be because the server's CPU does not support the AVX/AVX2 instruction set, or the CPU itself supports it but has been disabled by the system administrator. You can try contacting the system administrator to remove the restriction or change to a different server.
-
-References: https://github.com/opendatalab/MinerU/issues/591 , https://github.com/opendatalab/MinerU/issues/736
-
-
-### 8. Error when installing MinerU on CentOS 7 or Ubuntu 18: `ERROR: Failed building wheel for simsimd`
+### 2. Error when installing MinerU on CentOS 7 or Ubuntu 18: `ERROR: Failed building wheel for simsimd`
 
 The new version of albumentations (1.4.21) introduces a dependency on simsimd. Since the pre-built package of simsimd for Linux requires a glibc version greater than or equal to 2.28, this causes installation issues on some Linux distributions released before 2019. You can resolve this issue by using the following command:
 ```
-pip install -U magic-pdf[full,old_linux] --extra-index-url https://wheels.myhloli.com
+conda create -n mineru python=3.11 -y
+conda activate mineru
+pip install -U "mineru[pipeline_old_linux]"
 ```
 
 Reference: https://github.com/opendatalab/MinerU/issues/1004
-
-### 9. Old Graphics Cards Such as M40 Encounter "RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED"
-
-An error occurs during operation (cuda):
-```
-RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmStridedBatchedEx(handle, opa, opb, (int)m, (int)n, (int)k, (void*)&falpha, a, CUDA_R_16BF, (int)lda, stridea, b, CUDA_R_16BF, (int)ldb, strideb, (void*)&fbeta, c, CUDA_R_16BF, (int)ldc, stridec, (int)num_batches, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)
-```
-Because BF16 precision is not supported on graphics cards before the Turing architecture and some graphics cards are not recognized by torch, it is necessary to manually disable BF16 precision.
-Modify the code in lines 287-290 of the "pdf_parse_union_core_v2.py" file (note that the location may vary in different versions):
-```
-if torch.cuda.is_bf16_supported():
-    supports_bfloat16 = True
-else:
-    supports_bfloat16 = False
-```
-Change it to:
-```
-supports_bfloat16 = False
-```
-Reference: https://github.com/opendatalab/MinerU/issues/1508
\ No newline at end of file

docs/FAQ_zh_cn.md

 # 常见问题解答
 
-### 1.在较新版本的mac上使用命令安装pip install magic-pdf\[full\] zsh: no matches found: magic-pdf\[full\]
-
-在 macOS 上，默认的 shell 从 Bash 切换到了 Z shell，而 Z shell 对于某些类型的字符串匹配有特殊的处理逻辑，这可能导致no matches found错误。
-可以通过在命令行禁用globbing特性，再尝试运行安装命令
-
-```bash
-setopt no_nomatch
-pip install magic-pdf[full]
-```
-
-### 2.使用过程中遇到_pickle.UnpicklingError: invalid load key, 'v'.错误
-
-可能是由于模型文件未下载完整导致，可尝试重新下载模型文件后再试
-参考：https://github.com/opendatalab/MinerU/issues/143
-
-### 3.模型文件应该下载到哪里/models-dir的配置应该怎么填
-
-模型文件的路径输入是在"magic-pdf.json"中通过
-
-```json
-{
-  "models-dir": "/tmp/models"
-}
-```
-
-进行配置的。
-这个路径是绝对路径而不是相对路径，绝对路径的获取可在models目录中通过命令 "pwd" 获取。
-参考：https://github.com/opendatalab/MinerU/issues/155#issuecomment-2230216874
-
-### 4.在WSL2的Ubuntu22.04中遇到报错`ImportError: libGL.so.1: cannot open shared object file: No such file or directory`
+### 1.在WSL2的Ubuntu22.04中遇到报错`ImportError: libGL.so.1: cannot open shared object file: No such file or directory`
 
 WSL2的Ubuntu22.04中缺少`libgl`库，可通过以下命令安装`libgl`库解决：
 
@@ -39,59 +10,14 @@ sudo apt-get install libgl1-mesa-glx
 
 参考：https://github.com/opendatalab/MinerU/issues/388
 
-### 5.遇到报错 `ModuleNotFoundError : Nomodulenamed 'fairscale'`
-
-需要卸载该模块并重新安装
-
-```bash
-pip uninstall fairscale
-pip install fairscale
-```
-
-参考：https://github.com/opendatalab/MinerU/issues/411
 
-### 6.在部分较新的设备如H100上，使用CUDA加速OCR时解析出的文字乱码。
-
-cuda11对新显卡的兼容性不好，需要升级paddle使用的cuda版本
-
-```bash
-pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/
-```
-参考：https://github.com/opendatalab/MinerU/issues/558
-
-### 7.在部分Linux服务器上，程序一运行就报错 `非法指令 (核心已转储)` 或 `Illegal instruction (core dumped)`
-
-可能是因为服务器CPU不支持AVX/AVX2指令集，或cpu本身支持但被运维禁用了，可以尝试联系运维解除限制或更换服务器。
-
-参考：https://github.com/opendatalab/MinerU/issues/591 ， https://github.com/opendatalab/MinerU/issues/736
-
-### 8.在 CentOS 7 或 Ubuntu 18 系统安装MinerU时报错`ERROR: Failed building wheel for simsimd`
+### 2.在 CentOS 7 或 Ubuntu 18 系统安装MinerU时报错`ERROR: Failed building wheel for simsimd`
 
 新版本albumentations(1.4.21)引入了依赖simsimd,由于simsimd在linux的预编译包要求glibc的版本大于等于2.28，导致部分2019年之前发布的Linux发行版无法正常安装，可通过如下命令安装:
 ```
-pip install -U magic-pdf[full,old_linux] --extra-index-url https://wheels.myhloli.com
+conda create -n mineru python=3.11 -y
+conda activate mineru
+pip install -U "mineru[pipeline_old_linux]"
 ```
 
 参考：https://github.com/opendatalab/MinerU/issues/1004
-
-### 9. 旧显卡如M40出现 "RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED"
-
-在运行过程中（使用CUDA）出现以下错误：
-```
-RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmStridedBatchedEx(handle, opa, opb, (int)m, (int)n, (int)k, (void*)&falpha, a, CUDA_R_16BF, (int)lda, stridea, b, CUDA_R_16BF, (int)ldb, strideb, (void*)&fbeta, c, CUDA_R_16BF, (int)ldc, stridec, (int)num_batches, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)
-```
-由于Turing架构之前的显卡不支持BF16精度，并且部分显卡未能被PyTorch正确识别，因此需要手动关闭BF16精度。
-
-请找到并修改`pdf_parse_union_core_v2.py`文件中的第287至290行代码（注意：不同版本中位置可能有所不同），原代码如下：
-```python
-if torch.cuda.is_bf16_supported():
-    supports_bfloat16 = True
-else:
-    supports_bfloat16 = False
-```
-将其修改为：
-```python
-supports_bfloat16 = False
-```
-
-参考：https://github.com/opendatalab/MinerU/issues/1508

docs/output_file_en_us.md

 ## Overview
 
-After executing the `magic-pdf` command, in addition to outputting files related to markdown, several other files unrelated to markdown will also be generated. These files will be introduced one by one.
+After executing the `mineru` command, in addition to outputting files related to markdown, several other files unrelated to markdown will also be generated. These files will be introduced one by one.
 
 ### some_pdf_layout.pdf
 
-Each page layout consists of one or more boxes. The number at the top left of each box indicates its sequence number. Additionally, in `layout.pdf`, different content blocks are highlighted with different background colors.
-
+Each page's layout consists of one or more bounding boxes. The number in the top-right corner of each box indicates the reading order. Additionally, different content blocks are highlighted with distinct background colors within the layout.pdf.
 ![layout example](images/layout_example.png)
 
-### some_pdf_spans.pdf
+### some_pdf_spans.pdf(Applicable only to the pipeline backend)
 
 All spans on the page are drawn with different colored line frames according to the span type. This file can be used for quality control, allowing for quick identification of issues such as missing text or unrecognized inline formulas.
 
 ![spans example](images/spans_example.png)
 
-### some_pdf_model.json
+### some_pdf_model.json(Applicable only to the pipeline backend)
 
 #### Structure Definition
 
@@ -117,13 +116,39 @@ The format of the poly coordinates is \[x0, y0, x1, y1, x2, y2, x3, y3\], repres
 ]
 ```
 
+### some_pdf_model_output.txt (Applicable only to the VLM backend)
+
+This file contains the output of the VLM model, with each page's output separated by `----`.  
+Each page's output consists of text blocks starting with `<|box_start|>` and ending with `<|md_end|>`.  
+The meaning of each field is as follows:  
+- `<|box_start|>x0 y0 x1 y1<|box_end|>`  
+  x0 y0 x1 y1 represent the coordinates of a quadrilateral, indicating the top-left and bottom-right points. The values are based on a normalized page size of 1000x1000.
+- `<|ref_start|>type<|ref_end|>`  
+  `type` indicates the block type. Possible values are:
+  ```json
+  {
+      "text": "Text",
+      "title": "Title",
+      "image": "Image",
+      "image_caption": "Image Caption",
+      "image_footnote": "Image Footnote",
+      "table": "Table",
+      "table_caption": "Table Caption",
+      "table_footnote": "Table Footnote",
+      "equation": "Interline Equation"
+  }
+  ```
+- `<|md_start|>Markdown content<|md_end|>`  
+  This field contains the Markdown content of the block. If `type` is `text`, the end of the text may contain the `<|txt_contd|>` tag, indicating that this block can be connected with the following `text` block(s).
+  If `type` is `table`, the content is in `otsl` format and needs to be converted into HTML for rendering in Markdown.
+
 ### some_pdf_middle.json
 
 | Field Name     | Description                                                                                                    |
-| :------------- | :------------------------------------------------------------------------------------------------------------- |
+|:---------------| :------------------------------------------------------------------------------------------------------------- |
 | pdf_info       | list, each element is a dict representing the parsing result of each PDF page, see the table below for details |
-| \_parse_type   | ocr \| txt, used to indicate the mode used in this intermediate parsing state                                  |
-| \_version_name | string, indicates the version of magic-pdf used in this parsing                                                |
+| \_backend      | pipeline \| vlm, used to indicate the mode used in this intermediate parsing state                                  |
+| \_version_name | string, indicates the version of mineru used in this parsing                                                |
 
 <br>
 
@@ -324,7 +349,20 @@ First-level block (if any) -> Second-level block -> Line -> Span
             ]
         }
     ],
-    "_parse_type": "txt",
+    "_backend": "pipeline",
     "_version_name": "0.6.1"
 }
 ```
+
+
+### some_pdf_content_list.json
+
+This file is a JSON array where each element is a dict storing all readable content blocks in the document in reading order.  
+`content_list` can be viewed as a simplified version of `middle.json`. The content block types are mostly consistent with those in `middle.json`, but layout information is not included.  
+
+Please note that both `title` and text blocks in `content_list` are uniformly represented using the text type. The `text_level` field is used to distinguish the hierarchy of text blocks:
+- A block without the `text_level` field or with `text_level=0` represents body text.
+- A block with `text_level=1` represents a level-1 heading.
+- A block with `text_level=2` represents a level-2 heading, and so on.
+
+Each dict contains the `page_idx` field, indicating the page number (starting from 0) where the content block resides.
\ No newline at end of file

docs/output_file_zh_cn.md

 ## 概览
 
-`magic-pdf` 命令执行后除了输出和 markdown 有关的文件以外，还会生成若干个和 markdown 无关的文件。现在将一一介绍这些文件
+`mineru` 命令执行后除了输出 markdown 文件以外，还可能会生成若干个和 markdown 无关的文件。现在将一一介绍这些文件
 
 ### some_pdf_layout.pdf
 
-每一页的 layout 均由一个或多个框组成。 每个框左上脚的数字表明它们的序号。此外 layout.pdf 框内用不同的背景色块圈定不同的内容块。
+每一页的 layout 均由一个或多个框组成。 每个框右上角的数字表明它们的阅读顺序。此外 layout.pdf 框内用不同的背景色块圈定不同的内容块。
 
 ![layout 页面示例](images/layout_example.png)
 
-### some_pdf_spans.pdf
+### some_pdf_spans.pdf(仅适用于pipeline后端)
 
-根据 span 类型的不同，采用不同颜色线框绘制页面上所有 span。该文件可以用于质检，可以快速排查出文本丢失、行间公式未识别等问题。
+根据 span 类型的不同，采用不同颜色线框绘制页面上所有 span。该文件可以用于质检，可以快速排查出文本丢失、行内公式未识别等问题。
 
 ![span 页面示例](images/spans_example.png)
 
-### some_pdf_model.json
+### some_pdf_model.json(仅适用于pipeline后端)
 
 #### 结构定义
 
@@ -117,13 +117,39 @@ poly 坐标的格式 \[x0, y0, x1, y1, x2, y2, x3, y3\], 分别表示左上、
 ]
 ```
 
+### some_pdf_model_output.txt(仅适用于vlm后端)
+
+该文件是vlm模型的输出结果，使用`----`分割每一页的输出结果。  
+每一页的输出结果一些以`<|box_start|>`开头，`<|md_end|>`结尾的文本块。  
+其中字段的含义：  
+- `<|box_start|>x0 y0 x1 y1<|box_end|>`  
+    其中x0 y0 x1 y1是四边形的坐标，分别表示左上、右下的两点坐标，值为将页面缩放至1000x1000后，四边形的坐标值。
+- `<|ref_start|>type<|ref_end|>`  
+  type是该block的类型，可能的值有：
+  ```json
+  {
+      "text": "文本",
+      "title": "标题",
+      "image": "图片",
+      "image_caption": "图片描述",
+      "image_footnote": "图片脚注",
+      "table": "表格",
+      "table_caption": "表格描述",
+      "table_footnote": "表格脚注",
+      "equation": "行间公式"
+  }
+  ```
+- `<|md_start|>markdown内容<|md_end|>`  
+    该字段是该block的markdown内容，如type为text，文本末尾可能存在`<|txt_contd|>`标记，表示该文本块可以后后续text块连接。
+    如type为table，内容为`otsl`格式表示的表格内容，需要转换为html格式才能在markdown中渲染。
+
 ### some_pdf_middle.json
 
-| 字段名         | 解释                                                               |
-| :------------- | :----------------------------------------------------------------- |
+| 字段名            | 解释                                        |
+|:---------------|:------------------------------------------|
 | pdf_info       | list，每个元素都是一个dict,这个dict是每一页pdf的解析结果，详见下表 |
-| \_parse_type   | ocr \| txt，用来标识本次解析的中间态使用的模式                     |
-| \_version_name | string, 表示本次解析使用的 magic-pdf 的版本号                      |
+| \_backend      | pipeline \| vlm，用来标识本次解析的中间态使用的模式         |
+| \_version_name | string, 表示本次解析使用的 mineru 的版本号             |
 
 <br>
 
@@ -323,7 +349,14 @@ para_blocks内存储的元素为区块信息
             ]
         }
     ],
-    "_parse_type": "txt",
+    "_backend": "pipeline",
     "_version_name": "0.6.1"
 }
 ```
+
+### some_pdf_content_list.json
+
+该文件是一个json数组，每个元素是一个dict，按阅读顺序平铺存储文档中所有可阅读的内容块。  
+content_list可以看成简化后的middle.json，内容块的类型基本和middle.json一致，但不包含布局信息。  
+需要注意的是，content_list中的title和text块统一使用text类型表示，通过`text_level`字段来区分文本块的层级，不含`text_level`字段或`text_level`为0的文本块表示正文文本，`text_level`为1的文本块表示一级标题，`text_level`为2的文本块表示二级标题，以此类推。  
+每个dict包含`page_idx`字段，表示该内容块所在的页码，从0开始。
\ No newline at end of file